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ABSTRACT 

This policy guide argues the need for improved 
indicators of student coursework, illustrates the information that 
different types of measures convey, outlines the key choices that 
need to be made abont data collection, and suggests ways in which 
states might develop more valid and useful indicators of student 
coursework. In its broadest form, coursework includes the following: 
(1) the courses that schools offer; (2) patterns of coursetaking by 
different types of students; (3) the content of these courses; (4) 
their objectives; (5) instructional strategies; and (6) teachers* 
qualifications and experience. Examples provided are drawn from the 
School Reform Assessment (SRA) project, a 2-year exploratory study 
undertaken in California and Georgia to design a variety of high 
school coursework indicators. The SRA focuses on expanding and 
refining the technical quality of existing coursework indicators and 
accommodating the information needs of policymakers by providing 
indicators that measure the effects of major curriculum policies, 
valid coursework indicators must -neet the following four criteria: 

(1) they must be linked to a larger model of the schooling process; 

(2) they must differentiate among tracks or levels of the curriculum; 

(3) they must distinguish between the curriculum as it is intended by 
designers/policymakers and as it is actually implemented in 
schools/classrooms; and (4) they must measure the depth and breadth 
of the curriculum. Statistical data are presented in three tables and 
nine graphs. An appendix describes the SRA sample. A 28-itera list of 
references is included. (SLD) 
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PREFACE 



This policy guide aigues the case for improved indicators of student coursework, illus- 
trates the kinds of information they convey, and assesses tiie feasibility of altranative data collec- 
tion strategies fox producing them. It is not a typical reseaidi report sturanaiizing study findings 
in full detail, nor is it a handbook for technical experts charged witii designing new indicators. 
Rather, tiie guide is addressed to members of the policy community who \ise the information 
gKierated by state data systems and who must dedde on tfie value ar»d feasibility of gatiiering 
additional data. A companion to this vdtmie. The Design (^Imjmmed Qmrsework indicators: First 
Steps, Center for Res^irdi on Evaluatioiv Standards, and Student Testing (CRESST), UCLA 
Graduate School of Education, explores the technical issues involved in designing coursework 
indicators and is primarily in^ded to be used by those who develop new measures and manage 
data collection, analysis, and reporting. 

This policy gidde was prepared as part of a project sponsored by CRESST and funded by 
the Office of Educational Research and Improvement (OERI), US. Department of Education. 



SUMMARY 



What is taught, who teach^, and how they teach lie at the teart of schooling. Yet the de- 
velopment of improved curriculum measure has lagged far behind that of school resoiure and 
student achievement measures* Current indicators are inadequate and will become even more so 
as cunicular policy and practice evolve 



Coursework is the most tangible f^tiire of curriculiun at the high school WHAT IS 

level Althou^ coursework does not ^ccmtpass the full range of ideas, COURSEWORK? 

values, and knowledge that schods attempt to inculcate in students, it 

does capture essential elements of curriculum that can be m^sured in a 

systematic way. In its broadest form, coursework includes the a>urses 

that schools offer, patterns of coiusetaking by different types of students, 

the cratent of those coiuses, coursework objectives, the instructional 

strategies used/ and teachers' qualifications and experience. 

This policy guide argues the case for improved indicators of student 
coursework, illustrates the information different types of mmures con- 
vey, and outlines the key choices that need to be made about data collec- 
tion strategies. The ecamples used in the t^ are drawn from the School 
Reform Assessment (SRA) project, a two-year exploratory study under- 
taken to design a variety of coursework indicators. 



The need for better coursework indicators can be argued on both educa- 
tional research and policy grounds. Research on the relationship be^ 
tween student achievement and coiusework has shown that data on stu- 
dent test scores are of little use unless information is also available on 
factors that influeni^ the distribution of those scor^ and help shape 
changes in achievement, for example, the curricular opportunities af- 
forded diff^^t types of students. But that information must consist of 
more than course offerings and enrollment statistics enumerated simply 
by course title. Many course titles convey no information about content 
or how that content is presented. Because the curriculum offered within 
most high schools is differentiated according to student ability levels, 
and curricula also vary significantly acrc^ schools, course titles tell veiy 
little about what the ^ools are teaching or how coiirsework is 
influencing student performance. 

Over the past decade, elected officials, especially at the state level, have 
extended their traditional concern about how schools are governed and 
financed to include what the schools teach, who teaches it, and in some 
cases, how it is taught. Policymakers are now interrated in learning 
about the content students are receiving, and many have expressed con- 
cern about the effects of recent coursework iK>lides on low-achieving 



THE CASE FOR 
IMPROVED 
INDICATORS 
OF STUDENT 
COURSEWORK 



students and on the range of curricular offierings. Current data^ based on 
simple enrollment counts, is in^equate to answer their questions. 
Indicator designers will increasingly have to take into acxxnmt both the 
broader schooling context in which curriculum is delivered to students 
and the qu^tion of depth as wdl as breadth of coverage. Even states 
with wellnlevdoped indicator systems lack data beyond coiuse enroll- 
ment statistics, and more sophisticated data are limited to national sam- 
ples. 



DESIGN 

STANDARDS 
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INDICATORS 



Indicators that validly meas\ire which courses schools offer, the content 
included in those courses, die qualifications of the people teaching them, 
and the students enrolled in them must meet four technical criteria: 

• They must be linked to a larger model of the schooling pro- 
cess* 

• They must differentiate among tracks or levels of the 
curriculum. 

• They must distinguish between the curriculum as it is in- 
tended by designers and poliqnnakers and as it is actually 
implemented in schools and classrooms. 

• They must measure, to the extent possible, the depth of the 
curriculum as well as its breadth. 

Coursework indicators, however, must meet more than just technical 
standards* To be policy*relevant, they must also provide useful infor- 
mation for policy decisionmaking, and they must be feasible to collect 
and report* 



SAMPLE 

COURSEWORK 

INDICATORS 



Topic Coverage 



Student coursework consists of four basic elements: content or topic 
cov^ge, instructional strategies, curricular objectives, and teacher 
qualifications. Valid coursework indicators can help gauge how schools 
perform on each of these dimensions* 

Coursework patterns and trends can be klentified by asking teachers to 
note how many class periods they devoted to a particular topic in a spe- 
cific section and how they treated that topic. Topic coverage data can be 
used in a variety of ways. Reports of the average number of periods 
across classrooms and schools, for example, enable comparisons of the 
proportion of class time spent on formal institutions in U.S. government 
classes as oppc^ed to the proportion spent on political dynamics and 
policy outputs* Or they can be used to examine how the relative empha- 
sis accorded different topics compares with state curricultun frameworks 
and various professional standards, or how this emphasis has changed 
over time. 
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Qne meastire of the quality of a high school curriculum is the amount of 
an academic subjects erne content that is covered in a typical onuse and 
whether that proportion varira significantly acro^ sections of the same 
course. Sophisticated indicators can be constructed from topic a>veTage 
data to show the proporticmate coverage of core topics in different types 
of classes. For example^ data from a small sample of sdioc^ in the SRA 
project showed that while students in a typical algebra I section had 
more exposure to a core set of algebra and enriched topics than their 
counterparts in either pre-algebra or goieral mathematics classes, con- 
tent coverage varied enormously across sections of courses with the 
same title. Exduding &ctreme ^^du^, the proportion of topks covered 
ranged from 44 to 93 percent in algebra I sections; from 33 to 80 percent 
in pre-algdl>ni; and from 0 to 67 percent in goieral mathematics sections* 
In othar words, topic coverage data revealed that in this sample of 
schools, some algebra I students receive no more exposure to algebra 
content than students auolled in general mathematics and pre-algebra 
classes* 

Information about the mode in which content is being delivered to stu- 
dents can be obtained by asking about the frequoicy with which teachers 
use different instructional strategies (e.g., lecturing to the class, having 
students work in small groups, assigning rra^rch papers, ete.). 
Information about instructional strategies can provide important insights 
into the curricula that diff^^t types of students are receiving. For ex- 
ample, while the teacher lectured almost every day in US. history and 
government classes of all ability levels in the indicator design sample, 
more than twi« as many high- and averageability classes than low- 
ability classes required student piesentations at least monthly, and stu- 
dents in high-ability classes were three times as likely as those in low- 
ability classes to write research papas. 

The relative emphasis teachers place on different ob^ctives {e.g., devel- Curricular 
oping an attitude of inquiry vs. performing computations with speed Objectives 
and accuracy) is an indicator of their expectations for a particular course, 
and their choice of objectives is likely to influence how they configure 
topics and instructional activities within that course. Teacheis' reported 
objectives can thus suggest the direction in which coursework and 
teaching in a particular subject area may be heading. And, like topic 
coveitige and instructional activities, the relative emphasis accorded dif- 
ferent objectives also helps in differentiating among course levels-^.g., 
an overwhelming ma^rity of general mathematics teachers in the SRA 
sample reported a major emphasis on developing an awareness of the 
importence of mathematics in everyday life, while significantiy more 
al^ra I teachers listed cultivating a systematic approach to problem- 
solving as a major emphasis in their classes. 
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Teacher 
Qualifications 



A final hidi(3tOT of student coursew>rk is the match between teachers' 
quali&ations and thdr teaching assig^unents* States are inoea^ngly 
collecting teacher assignment data that can be linked to a database on 
tead^ certification status. However, rartification status in some sub- 
jects may provide only a partial, or ev^ a distroted, picture of teacher 
qualifications. For example^ 83 percent of the 310 nuthematics sections 
in the SRA sample were tau^t by teadiers certified in mathematics, but 
only 58 percent were taught by teachers who had majored in mathemat* 
ics or mathematics educaficm. Years of teaching experience and the re- 
cency of professional development in relevant subject matter are two 
other measure of teacher qualifications* 



DATA 

COLLECTION 

STRATEGIES 
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COURSEWORK 
INDICATORS 



The tradeoff between ensuring high technical quality of indicators and 
minimizhig cost and respondent burden becxnnes dearest when indicator 
system designers have to decide which data collection strategies to use. 
States and school districts that currently collect information on course- 
work typically use the least expensive and least burdensome option, that 
of recoitiing schooMevel ofiierings and enrollment categorized by con- 
ventional course titles* However, as the technioil limitations of these 
data grow more apparent and poliqrmakers continue to demand better 
ixtfbrmation, states will have to adopt data collection strategies that entail 
greater cost and respondent burden tmt produce more valid and useful 
information. Although in-depth methods such as classroom observa- 
tions will never be feasible for collecting routine indicator data, teacher 
surveys, combined with periodic collection of more detailed data, may be 
a reasonable alternative. 



In most subject areas, teacher surve)^ will require several years to de- 
sign, and each will take from 30 minutes to 1 hour to fill out. However, 
the costs of initial design work can be reduced if states and school dis- 
tricts form consortia to share responsibility for development efforts. 
Teacher burden am. be minimized by collecting coursework data only 
once every two or three years, by focusing on only several critical courses 
in each subject area, and by collecting data in cycles, for example, cover- 
ing English and social studies in one cycle and mathematics and science 
in the next. 

Ensuring the validity of the inferences drawn from indicator data re- 
quires that benchmark data be collected. These data are more difficult 
and costly to collect than routine indicator data, and they serve as stan- 
dards or anchors against which the validity of the routine data can be 
judged. Benchmark data require deeper probes than are possible with 
survey data and, as such, do not need to be collected as often or on as 
large a sample as conventional indicator data. 
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Benchmark data can serve four purposes in a state indicator 53rste]n: 

• ' Benchmark data can inform the development and evaliiation 

of new coursework measures because they represent the 
content and process of instruction much more ck^y than 
do more routine data (e.g^ in-<lepth intoviews with teachers 
or classroom oteervations \^ teadier sv^xveys), and they are 
less subject to the extraneous influences that limit or com- 
promise routine data. 

• After an indicator system has hem oj^rating for some time, 
bemhmark data can be used to assess the extent to which 
measures have been corrupted by social d^irability or pol- 
icy pressures (e.g., teach^ reporting classroom l«haviors 
consistent with rdarm policies, while their examinations do 
not reflect such behaviors). 

• Benchmark data can provide a context for interpreting 
trends or unexpected changes in aggregate indicator data 
(e.g., through sdtool-level case ^udies). 

• Benchmark data can be used to complement and enhance 
indicator system data through studies that explore in depth 
areas of policy or practitioner concern that indicator date 
have signaled as problematic (e.g*, curricular opportunities 
becoming more imequal in certain types of schools). 

We recommend that states interview small samples of district- and 
school-level staff, code student transcripts, arid review teachers' syllabi 
and course materials. We also recommend that states wishing to collect 
more extensive data on student coursework make significant and con- 
tinuing investments in benchmark data. About one-quarter of a state's 
coursework indicator budget should be allocated to benchmark data col- 
lection. 



Those who conuxUssion and design new indicators can rarely control 
how they are used. Nevertheless, taking into account how similar mea- 
sures have been used in the past and anticipating how new ones might 
be used could help to avoid problems. State officials should keep in 
mind, first, that as indicators gain significance in the policy world, they 
tend to provide less valid information about what is actually occurring in 
classrooms — that is, they are likely to become corrupted as measure 
Sea>nd, indicators are powerful levers for changing classroom behavior, 
but the chan^ are often not what policymakers and indicator designers 
intended. 
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Such problems can be minimized if state offidak are clear from the very 
beginning about the appropriate tises of coursework indicators, lliese 
uidioitors are best tised to docimient trends, assess the p^fbnnance of 
the overall educational sys^, identify and ta^et {KiUcy solutions^ and 
aid in local improvement efibrta If they are used to compare schods or 
to assess teacher performance, they will probably cease to provide valid 
information and may even distort instructional practice in ways that 
compromise student learning. 



CONCLUSIONS Improving the information base on which decisions are made about edu- 
cational policy and practice is not easy. Designing valid and useful mea- 
sures takes time; collecting the necessary data imposes financial and 
other costs; and using indicator data in a constructive way requires con- 
siderable thought and planning. While these r^resent formidable chal- 
lenges in any type of indicator development, Uiey are particularly de- 
manding in the case of couxsework indicators because of the need for 
validation and deeper probes in the f omt of benchmark data« 

Despite these challenges, the price that states will pay for not collecting 
better rouisework data may be tmacceptably high* Strategies for attain* 
ing performance goals ^nnot be implemented, classroom learning op- 
portimities cannot be equalized, and {x>Ucy impacts cannot be Actively 
monitored if current information ^ps persist. Indicator development 
will never have the visibility or politic appeal of new policy initiatives 
aimed at improving schools. Nevertheless, data about what schoobf are 
teaching form the comerstone of those initiatives, and the constituents of 
public schooling need valid infomiation on coursework to continue the 
work of educational reform. 
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1. THE NEED FOR IMPROVED INDICATORS 
OF STUDENT COURSEWORK 



While no single theme characterize the education reforms of the past five years, two pre- 
dominate The first is an emphasis on cuiriculimi: what students stiidy; what skills, concepts^ 
and knowledge are included in their course of study; and who teaches them. This emphasis re- 
sulted from growing concern that American students are not receiving as rigorous a curriculum 
as their coimterparts in nations that compete economically with the United States, and from evi- 
dence that learning opportunities are unequal across different types of local districts, schools, and 
students. Policy actions addressing these concerns have included increased coiase requirements 
for high school graduation; state and loral guidelines for core curricula; efforts to improve the 
quality of textbooks; and greater attention to the match between teacher qualifications and teach- 
ing assignments. 



The second theme is a demand for better information about the schooling THE D^iAN P 
process and its outcomes. This demand is premised on the beli^ that FOR BETTER 
schools should be held accountable to the public and to their elected rep- INFORMATION 
resentatives and that soiud information about educational conditions 
and perfonnance is critical to improving the quality of schooling. 
Policies designed to meet this need have included the expansion of state 
testing programs; the upgrading of state indicator systems to encompass 
measures beyond student scores on standardized t^ts; and a variety of 
programs that reward, pimish, or assist schools based on indicator data. 

This policy guide joins the two themes by suggesting ways in which 
states n\ight develop more valid and useful indicators of student 
coursework. At the high school level, coursework is the most tangible 
feature of curriculum, defined here as the complete array of learning op- 
portxmities provided to students throughout their years of schooling. 
Although the notion of coursework does not encompass the full range of 
ideas, values, and knowledge that schools attempt to inculcate in stu- 
dents, it does capture most of the essential elements of cuniculimi that 
can be measured in a systematic way* In its broadest form, coursework 
includes the rourses that schools offer, patterns of coiusetaking by dif- 
ferent types of students, the content of those courses, coursework objec- 
tives, the instructional strategies used, and teachers' qualifications and 
experience.^ 



^This report focuses on hl^ sdtool curricula because notion of coursework is 
well-defined at that level Curricular contrat in elementaiy schools is not as dearly divided 
into separate courses, and instructional groupings operate at the level of the work group 
rather than the entire dassTDom; Ihei^re, curriculum indicators need to be conceptualized 
and misused differently at that leveL Section 6 discusses Aese distinctions and their 
impUcaCions for designir^ indicators of the curricuhim presented to different types of 
elementary school students. 
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The strategies and recommendations discussed here are based on an ex- 
ploratory project undertaken to develop reliable coursework indicators 
that are also: 

• S^isitive to major policy changes, 

• Relevant to policymakers' information needs, and 

• Efficient to collect and r^rt. 

This guide does not outline step-by-step procedures for designing new 
indicators. Not only was our researdi too preliminary to produce a doc- 
ument of such certainty, but each state will have to tailor its indicators to 
its own polid^, instructional practices, and information needs. 

Although the design strategies we present are suggestive, we argue 
strongly that indicators of student coursework are needed and that cur- 
rent measures are inadequate and will become even more so as curricular 
policy and practice continue to evolve over tl'ie next decade. The re- 
mainder of this section presents a case for improved indicators of student 
coursework Section 2 discusses the characteristics of valid and useful 
coursework indicators. Section 3 briefly simunarizes the research project 
on which this guide is based. Section 4 presents sarriple coursework 
indicators and illustrates the kinds of information they can provide* 
Section 5 assesses different strategies for collecting data on student 
coursework Finally, Section 6 outlines steps that states can take to de- 
velop improved cotusework indicators. 



WHY DEVELOP 
IMPROVED 
INDICATORS 
OF STUDENT 
COURSEWORK? 

The Link 
Between 
Coursework and 
Student 
Achievement 



The reed for improved coursework indicators can be substantiated on 
both educational research and policy grounds, and argiunents for their 
development are reinforce by the limits of existing indicators. 

Research studies over the past fifteen years have documented the empir- 
ical relationship between student achievement, the types of courses 
taken, and the content and level of those cotirses. Raizen and Jones 
(1985) summarized four studies based on nationally representative stu- 
dent samples that showed a strong correlation between the niunber of 
mathematics courses students take and their achievement in mathemat- 
ics^ This relationship persists even when backgrotmd variables such as 
home arui community enviromxi^t and previous mathematics leanting 
are taken into aca>imt. Research has cilso shown that the level, as well as 
the number, of coiirses students take is correlated with achievCTtent. 
Jones et aL (1986), after controlling for socioeconomic status (S^) and 
test scores two years earlier, found that stud^ts in the High School and 
Beyond (HS&B) sample with at least five transcript credits in mathemat^ 
ics cour^ at or above the algebra I level scored an average of 17 per- 
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centage points higher on a standardized mathematics test than those 
with no course credits in higher-level mathematics. 

Some of the most compelling evidence about the relationship between 
achievement and curricular content comes from the Second International 
Math^natics Study (SIMS), which attempted to e>qplain the poor p^- 
formance of US. students relative to that of ^dents in otl^ indus- 
trialized countries. The SIMS researchers teg^n by eliminating what 
they called ''deceptive explanations." For example, they fouiui that 
althou^ the Japanese sdiool y ^ is longer than tiie VS. sdtool year (243 
days vs. 180), Japanese students spend less time in mathematics 
instruction than thdr American countoparts (101 hotus p^ year vs. 144). 
After eliminating the deceptive explanations, the researchers found 
real — and striking— differences between the ways curricula are 
organized in the hi^est-achieving countries and the way they are 
organized in the United States. At the lower-secondary level the 
Japanese curriculum ^phasizes algebra; the curricula in FrarKe and 
Belgium are dominated by geometry and fractions. In contrast^ VS. 
schools allocate their curricula more equally across a variety of topics — 
thus covering each stibject much more superficially. The mathematics 
curriculum in VS. schools is characterized by extensive repetition and 
review, and little ir.tensity of coverage. This low-intensity coverage 
means that individual topics are treated in only a few class periods, and 
concepts and topics are quite fragmented (McKitight et al., 1987; 
Burstein, forthcoming). 

Past research, most notably the SIMS, has product two important find- 
ings for indicator development First, research has shown that data on 
student test scores are of little use urdess information is also available on 
the factors that influence the distribution of those scores across different 
types of students and schools and that help shape changes in achieve- 
ment over time. Test-score data can indicate whether educational condi- 
tions are getting better or worse, but they provide little insight into why 
particular trends exist or how problems could be solved or successes 
replicated. For example, the most recent National Assessment of 
Educational Progress (NAEP) fotmd that student gains in matheirtatics 
achievement since 1978 have been confined primarily to lower-order 
skills, and that only about half of all 17-year-olds have mastered mathe- 
matical procedures such as solving simple linear equations or making 
decisions based on information drawn from graphs (Dc^sey et aL, 1988). 
That information, however, leaves some very basic questions unan- 
swered: What kinds of students are most likely to lack these skills? Is 
the content on which students are being tested offered in most high 
school mathematics courses? Are the teachers who teach those cour^ 
adequately prepared to convey the relevant content? 
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supplemefftb$g 
tesUscore data with 
information limited 
to course offerings 
and mroUments 
will not remedy ^ 
current 

inadequacies of 
most indicator 
systems. 



Indicator data on student outa>mes document problrans that need to be 
addressed; data on schooling trends, e.gv in coiusework, help identify 
the soxuces of Uie problems and point to potottial solutions. 

The second important research finding is that supplementing test-score 
data with information limited to the titles of course offerings and eiuoU* 
meats will not remedy the current inadequacies of most indicator sys* 
terns* Data on the types of courses that schools offer and tite niunb^ of 
students enrolled in them provide only the most superficial understand- 
ii^ of the relationship between curriculum and achievement As the 
SIMS results illustrate^ the essmtial information is not just whether stu- 
dents take a particular course or even a sequence of coiuses, but rather 
what content is covered in those courses and how that content is pre- 
sented. Measuring coursework solely by titles of offerings and numbers 
of ^irollments produces confusing information that is of little use to ei- 
ther educational policy or instructional practice. 

One of the major reasons for the lack of information conveyed by course 
titles alone is the differentiation of curricula within high schools accord- 
ing to student ability levels. Most comprehensive high schools have 
traditionally been characterized by multiple tracks^ a system that has 
been shown to have dear consequences for the learning opportunities ac- 
corded different types of students (Oakes et al., 1990; Oakes, 1985; 
Gamoran and Berends, 19S7; Gamoran, 1987). In some subject areas, 
particularly mathematics and science, such differences are partly cap- 
tured in course titles or in tl«e level designations appended to those titles. 
However, in other subject areas, such as sodal studies, course titles tell 
very little about the breadth or depth of content coverage or the instruc- 
tional strategies used. The information problem is further confounded 
by the movement of some high schools away from tracking to heteroge- 
neous classes while others maintain the traditional system. 

Basing coursework measiues on course titles will become even more 
problematic as curricular approach^ change over the next decade. 
Professional groups such as the National Council of Teachers of 
Mathematics (NCl'M) are recommending that mathematics courses and 
topics be reconfigured (e.g., they should more closely integrate ideas 
from algebra and geometry) as a way of strengthening high school math- 
ematics insbuction (NCTM, 1989). At the same time, some schools and 
districts have lengthened the traditional year-long algebra I sequence to 
two years to provide lower-achievmg students with exposure to algebra. 
In a state where either of the above policies has been implemented, the 
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titles of course offorings and enrollments will provide little infbnnation 
that is comparable over time or place* 

Past research provides dear evidence to support the argument that indi- 
cator systems must include measures of student coursework* Research 
findings, coupled with the changing nature of curricula, also suggest that 
coursework indicators must more soph^cated than simple coimts of 
offerings and enrollments. 

The call for improved indicators of student coiusework is based not only 
on educational research^ but also on the information needs of policymak- 
ers* More than at any other time in recent history, policymakers at all 
governmental levds are asking questioi^ about public schools and tak* 
ing actions that require more and better data* This interrat in improved 
indicators is reflected at the federal level in efforts to expand and en- 
hance the quality of national and international data; at the state level, in 
the work of organizations such as the Council of Chief State Sch<K)l 
Officers (CCSSO) to strengthen the capacity of states to collect and report 
indicator data and to standardize selected information acniss states; and 
at the district level in the activities of individual states and local districts 
to track schooling conditions and monitor the progress of their educa- 
tional reforms. 

The growing demand for better coursework indicators stems from the 
central fact that over the past decade, elected officials, especially at the 
state level, have extended their traditional concern about how schools are 
governed and financed to include what schools teach, who teaches it, 
and in some cases, how it is taught Hgure M depicts this expansion of 
the policy sphere. The upper circle represents that part of the educa- 
tional system most open to policy influence, the area of how schools are 
financed, governed, and organized The lower circle includes the core 
features of schooling that are more difficult for policy to iitfluence di- 
rectly (e.g., student characteristics, teaching quality, and instructional 
niethods). However, numy factors lie between these two realms (e.g., 
how teachers rre trained, wliat content students are taught). TTiese fac- 
tors affect the classroom directly but are also open to policy influences 
and have been the major targets of recent reform initiatives. Elements of 
schooling for which only practitioners needed information in the past are 
now also the purview of policymakers. 

Policymakers are not only interested in the coursework content that stu- 
dents are receiving, they have also expressed concern al )ut the effects of 
recent coursework policies on low-achieving students and about whether 
reforms have generated any unintended consequences, such as reduced 
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curricular offerings or increased dropout rates.^ They find that current 
data, based on simple enrollment coimts, are inadequate for answering 
their questions. 



...botft sets of 
requirements should 
lead indicator 
designers to take 
into account tite 
broader schooling 
cont&ct in which 
curriculum is 
delivered to 
students; how 
Oirriculum differs 
in content and 
treatment for 
different students; 
and depl^ as well 
as breads of 
coverage. 



This growing interest in monitoring the courses that different tyf^ of 
students are taking and in assessing the impacts of recent coursework 
policies has led pK>licymaker5 to demand information that is quite similar 
to what research suggests is important Both sets of requirements should 
lead indicator d^gners to take into accotmt the broader schooling con- 
text in which curriculum is delivered to students; how curriculum differs 



RafornipoUdM 




Rg. 1 .1— The expanilon of the echieattonal poltey system 



^Thcse information needs have been expressed by p<dicymakers in vaiious forums 
througlu^ut the country; they also emerged in a focus group session that we conducted 
with 20 governors' education aides and in telephone interviews with staff from national 
associations representing state policymakers as wel! as state and local policymakers and 
their staffs* 
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in content and treatment kff diferent students; and depth as well as 
breadth of coverage. Widi such data, ^ following kinds of policy ques- 
tions could be answered more systematically: 

• How do couisetaldng patterns otmpare in urban and rural 
sdiools? Among students of different ethnic groups? 
Between boys and girls? 

• Have state police that increase course requirements for 
high school graduation resulted in students taking more dif- 
ficult academic courses or just more lower-level ones? 

• To what extent is the course content that is suggested (or 
niandated) by the state reflected in individual schools and 
classrooms? 

• What is the match between teacher qualifications and their 
course assigiunents? In what courses are out-of-field as- 
sigiunents most prevalent? 

Even states with well-developed indicate systons, such as California, The Limits of 
lack data beyond course enrollment statistics. More sophisticated indica- Existing 
tor data are limited to national samples such as the National Education Indicators 
Longitudinal Study (NELS). California's school perfonxtance reports in- 
form the public about enrollment trends in selected upper-level high 
school courses and the proportion of students enrolled in those courses 
required for admi^on to the University of California. However, they do 
not track trends in lower-level courses, nor do they provide information 
on whether a>urse content is similar across schools or districts. 

Several recent studies of trends in student coursetaking illustrate what 
can be learned from coiuise enroUment data. These studies also indicate 
the limits of such informatioa A study conducted by Westat for the 
National Center for Education Statistics (NCES) compared the coursetak- 
ing patterns of 1987 high school graduates with those of students who 
completed high school in 1982. The data for the 1987 cohort were ex- 
tracted from approximately 15,000 transcripts from a nationally repre- 
sentative sample of high schoob selected for the 1985-86 NAEP and from 
12,000 transcripts of 1982 high school graduates who participated in the 
HS&B study. Westat found that 77 percent of the class of 1^7 had taken 
algebra I, as compared with 65 percent of the earlier cohort; the propor- 
tion of students taking biology had increased from 75 percent to 90 per- 
cent; and 71 percent of the 1987 graduates had taken at least one 
semester of Amcdcan government, compared with 57 percent of the class 
of 1982 (Bennett, 1988). Other analyses of these same date compared 



• K 21 

ERIC 



rates of coiusetaking by ethnidty, ^nder, and curricular track^ (Goertz, 
1989). 

These studies provide evidence that couxsetaking patterns in the imme- 
diate post-Nohon at Risk period were consistent with policymakers' ex*- 
pectations in raising course requirments for high school graduation. 
They also provide some information on coursetaking patterns across dif- 
ferent types of students— e.g., they reveal that the gap b^ween the pro- 
portions of white and black students taking algebra is ck^g about three 
times as fast as the gap in geometry (Goertz, 1989: Table 10)* 

But there are many imjwrtant questions that such data cannot answer. 
For example, tl^ do not indicate whether the a^^ebra I taken by the later 
cohort is e;>saitially the same course as that offered to the earlier group 
This is a signifiomt issue because some qualitative data (e.g*. Bureau of 
Research and Assessment, 1^) imiic^te that as more lower-ability stu- 
dents have been placed in algebra classes, teachers have had to cover 
fewer topics or cover them in less depth. However, other evidence sug- 
gests that the algebra I course may have become more rigorous as states 
and local school districts have moved to uj^rade course quality through 
curriculum reform and textbook adoption policies. The data also reveal 
nothing about the relationship between Westaf s classification of 
students into curricular tracks and their actual track as^gnmente and 
exposure to subject-matter content within their own schools* This 
measure of curricular differentiation may also have little meaning over 
time as more schools move toward heterogeneous classes. 

Other studies have examined trends in coursetaking on the basis of en- 
rollment statistics* Hanson (1989) used data collected by the Florida 
Department of Education to analyze change in coursetaking in 16 of the 
24 secondary schools in Dade County. These data provide information 
on the numl^ of sections of various course that each school off^ and 
the enrollment in those courses. Although such data do not provide 
studrat-level information, as transcript data do, they can indicate the dis- 
tribution of enrollment within and across subject areas as coursetaking 
patterns shift For example, Hanson found that mathematics enrollments 
had generally shifted toward less academically oriented courses (from 
algebra and computer applications to basic skills, informal geometry, 
and general mathematics). Across subject areas, sdence and foreign lan- 



*^estat classified students in the acidemk track if they earned at least 12 credits tn 
the core areas of English, histoiy and sodal studies, mathematics, and/or science and dkl 
not meet Uie requirements for me vocational track* Students who euiied at feast 3 credits 
in an occupationally specific vocational educaticm area and did iK»t meet the requirements 
for the academic track were classified in the vocational track. Students who did not meet 
the requirements for either tte academic or vocational track were classified as being in 
neither. This clarification system is independent of the assignment policies of individual 
schools and has no direct relationship to the content of tite onsrses students took. 
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guages gained largely at the ^cpense of vocational education, which lost 
19,000 students, or 65 percent of the enrolln^nt increase experienced by 
other departments. This information is undoubtedly very important in 
tracking policy ejects, but key pieces of data are missing. 

Efforts are now being made to expand coursework indicators to include 
measures of course content and instructional strat^es. However, such 
data are currently collected only on national samples and have not been 
incorporated into statewide indicator systems. In its teacher question- 
naire, the NELS asks teachers of each student in the survey to report on 
the topics covered in their classes and the emphasis accorded each topic, 
and on the materials and types of instructional activities used. 

Information is also requested about the ability level of each class. This 
information can relate course content to particular levels or tracks of the 
curriculum, yet it teUs little about the curriculum offered students acrc^ 
the omntiy. The NELS includes no questions about the amount of class 
time sp^t covering different topics, and no attempt is being made to 
validate the survey data with other data ct^ection techniques that probe 
more deeply into the actual process of classroom instruction. 

In arguing that coitrsework indicator need to measure more than course 
offerings and emvllments, we recognize that we are propc^g that states 
expand their data collection efforts considerably. For example, according 
to Blank and Schilder (1989), only 29 states report mathematics and 
science enrollments by course, and only 23 report on the proportion of 
teachers teaching out-of-field, as defined by the teachers' certification 
status. Although a number of state education agencies indicated that 
they were coitsidering collecting data on school- and classroom-level 
curriculum implementation (including reviews of school curriculiun 
outlines, teacher surveys, classroom observations, and "opportunity-to- 
leam" or topic-coverage questioiuiaires), these were viewed as 
"potential" rather than actual methods of data collection (Blank, 1988). 
Nevertheless, the imperatives of policy and practice continue to move 
states toward desigiUng and using more complete indicators of student 
coursework. 

The next section outlines standards of techiucal validity and policy use- 
fulness that should guide the design of coursework indicators. 
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2. DESIGN STANDARDS FOR COURSEWORK INDICATORS 



Educational indicators must meet technk^ criteria that ensure validity, and they must be 
policy^relevant, useful^ and feasible. This section outlines the d@ign standaids for couisework 
indicators. More general discussion of educational indicators can be foiwd in Oakes (1986), OERI 
State Accountability Study Group (1988), and Shavelson et aL (1989)* 



VAUD 

COURSEWORK 
INDICATORS 



Indicators that validly measure which courses schools the content 
included in those courses, and who teaches and enrolls in them should 
meet four technical ariteria: 



Th^ should be linked to a larger model of the schooling 
pro6ess* 

They should differentiate among tracks or levels of the cur- 
riculum. 

They should distinguish between the cumculiun intended 
by designers and policymakers and the one actually imple- 
mented in schools and classrooms* 

To the extent p<»sible, they should measure the depth of the 
curriculum as well as its breadth. 



Embed 
Couisework 
Indicators in a 
Model of 
Schooling 



Coursework indicators cannot be developed and used independently of 
indicators that measure the level and type of resources available to 
schools^ the quality of the teaching staff, and a variety of student*related 
measures such as promotion and dropout rates, postsecondary matricu- 
lation, and a broad range of achievement scores. These indicators need 
to be interpreted within the conte^ of a conceptual model of the way^ 
different components of the educational system relate to one another. 
Without such a model single indicators can easily be misinterpreted. 
The model may be simple and intuitive, or it can be more a>mplex. The 
important point is that it should identify the ncujor elements of the edu- 
cational s)^tem and illustrate the relationships among those elements. 
Such a model caxmot specify relationships in either a strictly predictive or 
a causal sense, but ii can serve as a framework, showing logical linkages 
among parts of the schooling system and correlational relationships 
supported by past research. An understanding of these relationships is 
particularly important in interpreting trends in coursework indicators, 
e.g., how changes in cotuisetaking patterns or content coverage relate to 
demographic trends in school enrollments or to shifts in achievement test 
scores. 
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Indicators that att^pt to describe curriculum without considering dis- 
tinctions among student ability levels will otecure crudal attributes of 
the system, such as which students have ao^ss to what types of learning 
throughout their academic careers. We have already dteatssed the im- 
portance of measurix^jyfhe extent to which cuzriculiun is di^erentiated 
by student ability ^^^ings. In some cases, those distinctions are clear 
from tile course tideS/ but in many others, differences aerosol levels are 
evident only if topic coverage and ins^ Honal strat^es are examined. 

Curriculum can be d^ned in several ways, from tiie id^ standards 
rendered by subject-niatter experts, to various state and local policies, to 
district ciuriculum giiides, to teacto* plans, to actual teach^-student in- 
teractions within classrooms (C^ikes and Carey, 19S9). Coursework indi- 
cators should characterize the implemented curriculum so that it can be 
compared with tite interuicd one (Mumane and Raizen^ 1988). Valid 
measure of the curriculum as it is actually presented in classrooms are 
needed to answer some very basic questions, e*g.. What proportion of 
high school courses meet or exceed the curricular standaids advocated 
by pro^sional organizations in sdoice, mathematics, or English? To 
what extent are state model curriculum framewoite reflected in teachers' 
content coverage and instructional strategies? 

One solution to the problem of coiuse titles providing inadequate infor- 
mation would be to measure the breadth of curricular content by deter- 
mining what topics are rovered in a particular omrse and the amount of 
time spent on each. However, as vrill be illustrated in Section 4, topic 
coverage in some subjects does nut adequately distinguish among 
courses. For example, a basic or remedial-level US. history course might 
devote as much time to the Civil War as an advanced or honors-level 
couise. What differentiates the two is the depth of coverage and the 
method of presentation. One course might stress discrete facts such as 
the dates of important battles^ and might rely heavily on classroom lec- 
tures and on reading a textbook in class, while another dass might focus 
on the broad sodal and economic issues raised by the Qvil War and em- 
phasize class discussions and independent student research. The depth 
of the ciuriculiun needs to l>e measui^ if states and local district are to 
have a robust set of coursework indicators. 
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If coursework indicators are to inform policy and practice, they should 
meet more than technical standards — they should also contribute to pol- 
icy decisions, and they should be feasible to collect and report 
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Create ITseftU 

Cottisework 

Indicators 



Couisework indicators should be sensitive to major policy changes and 
they should provide information on the factors ov^ which policy has 
some direct influence. 



Indicator data do not a^rd the level of detail and rigor needed for care- 
ful evaluation of individual policies or pn^rams (MacRae, 1985; 
Shavelson et aL, 19^). However^ such data can surest whether trends 
in the status of key educational indicators are consistent with what poli- 
cymakers hoped to achieve with particular types of policy. For example^ 
policymakers may have expected high^ graduation requirements to in- 
cr^ise both the amoimt and academic rigor of high school couisetaking; 
a comprehensive s^ of couisework indicates could indicate whether or 
not such chan^ had occurred but could not be used to ascertain 
whether th(^ changes were due to spedflc policy interventions or to 
other factors. In other words, indicator data cannot measure the inde- 
pendent effects of ^gle policies, but they can signal whether changes 
are consistent with broad policy initiative. 

As the scope of educational policymaking cx)ntinues to expand beyond 
the traditioital focus on how schools are financed and governed to con- 
cerns about what is taught and who teaches it, indicator data that can 
inform policy decisions will become an increasingly essential resource. 
However, because of the immediacy of the policy community's informa- 
tion demands, some data that would be useful to educators and re- 
searchers may have low priority. A major criterion for the inclusion of 
any particular coursework indicator in a policy-relevant system will be 
its ability to measure a factor over which policymakers have some direct 
influence. For example, while the curricular goals that teachers seek to 
promote or the nature of their interactions with individual students are 
important to a valid understanding of the schooling process, th^ fac- 
tors are far from the direct reach of policy. Therefore, when choices have 
to be made about which indicators to include in a state or local system, 
the highest priority should be given to thc^ that measure factors most 
directly influenced by policy. 

Emphasizing the information needs of policymakers and other non- 
experts, however, does not preclude obtaining information that can also 
be useful to educators and researchers. The most effective indicators are 
those that serve multiple purposes. Section 4 presents an example of 
how the same indicator can be reported in different formats and levels of 
detail to accommodate different audiences. However, not all indicators 
are equally useful to all audiences, and a single indicator system may not 
meet the diverse inforaiation needs of policymakers, educators, the gen- 
eral public, parents, and researchers. While informing all these audi- 
ences should be a goal of indicator design, resource and other constraints 
may preclude its achievement Therefore, indicators with explicit policy 
relevance should be given priority over factors less directly influenced by 
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policy: Satisfying the infcmnation demands of policymakers is a primary 
hmction of indicator systems (McDonnell^ 19S9), and indicator data are 
not an efficient source of more idio^nicratic information about individ- 
ual schools, classrooms, or students. 



The standards of technical validity and policy usefulness outlined above 
are ideals, and they must be tempered by the realities of policy and prac- 
tice. Some of these r^ties relate to cost— for example, measuring cur- 
ricular depth may require data a>llection methods that are too costly for 
widespread use by states and local districts. Some realities relate to the 
burden that can reasonably be imposed on schools for collecting indica- 
tor data and on state agendes for analyzing and interpreting them. 
Otiier constraints include tiie need to produce timely date on a schedule 
that is compatible with policymakers' decision cycles, and to report 
indicators that can be imderstood by a broad range of audiences- 
parents, policymakers, and newspaper readers, as well as educators. 

Throughout the remainder of this guide, as we present sample course- 
work indicators and assess different strategies for a>Uecting data on stu- 
dent coursework, we shall note the tradeoffe in cost and burden between 
less piedse or less valid indicators and more sophisticated ones. Hist, 
however, we briefly stmrntarize the res^rch project from which they are 
derived. 
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3- THE SCHOOL REFORM ASSESSMENT PROJECT 



The School Reform Assessment (SRA) project^ a two-year exploratory design study, had 
two goals: to expand upon and refine the technical quality of existing coursework indicators, and 
to accommodate the information needs of policymakers by providing indicators that would mea- 
sure, at least in a general way, the efiects of major curriculum policies. 



PURPOSE AND 
APPROACH 



These two ob^ctiv^ narrowed our task to designing measures of 
couisetaking that could be implemented by state governments as part of 
their existing information ^thering and indicator systems. We viewed 
our role as that of developing a template that states could adapt after 
more extensive field*testing, to their own policy concerns, information 
needs, and data collection procedures. Thus, we had to concentrate on 
measures for which data could be efficiently collected through siwey s of 
school administrators, teachers, and students. 



... titemafor 
contribution of 
ffce SRA project 
would not be in 
developing 
entirely new 
indicators, but in 
refining existing 
indicators, 
adapting them to 
the framework of 
state indicator 
systems, and 
validating them 
throu^ a number 
of benfAmarking 
procedures. 



We decided to draw primarily upon existing measures from sources such 
as the 5IMS-IEA study, NEL5, and NAEP. However, because the valid- 
ity of many questionnaire items typically used in such studio has not 
been tested, we also tmdertook several benchmarking procedures, in- 
cluding interviews with school- and district-levd personnel, evaluations 
of course materials, and the examination of student transcripts to verify 
data obtained from the surveys.^ The in-depth interviews and course 
materials provide information that is much closer to the actual content of 
instruction than that provided by enrollment statistics or even most sur- 
vey items; thus, they constitute evident about validity or the extent to 
which more routinely collected data are accurately tapping what schools 
really teach,^ Transcript analysis is an important source of historical data 
on how coursework patterns for different types of students have 
changed since the pre-reform period, and it provides a way of ascertain- 
ing whether the indicators we developed would be valid if the nature of 
the curriculum were to change significantly. Thus, we decided that the 
major contribution of the SRA project would not be in developing en- 
tirely new indicators, but in refining existing indicators, adapting them 
to the framework of state indicator systems, and validating them through 
a number of benchmarking procedures. 



Because of resource and time constraints, we focused on three mathemat- 
ics course cat^ories — mathematics below algebra I (e.g*, general math- 
ematics, consumer mathematics, pre-algebra), algebra I, and algebra II — 



^The role of benchmark data in a state indicator system is discussed at length in 
Sections. 

^Forthcoming report by Daniel Koretz, The RAND Corporation- 
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and two social studies courses-4J*S< history and U5. government 
These subjects were selected because they were particularly affected by 
changes in state high school graduaticm requirements. The specific 
coiirse categories were chosai because analyse of local responses to 
state curriculum policies sug^sted that these categories would capture 
the range of local effects (Oun^ White, and Patterson, 1989). Although 
this devdopment effort is limited to five coiuse categories, we believe 
that it can serve as a prototype for other subjects and cotirses. 



This study was a)nducted in two states, California and Georgia, to ccm* 
trol for the policy contact in which indicates would be developed and 
used* By taking into account state policies, we could develop indicators 
that would useful for assessing how the courseworic delivered in local 
hig^ schools compares with ^te curricular objectives. These two states 
were selected because data on their recent policies and local responses to 
those policies were available from earlier research. 

As noted in Section 1, California's indicator system is one of the most 
highly developed state systezns in the cotmtry, but its information on 
student coursetaking is limited to school-level enrollmri\t statistics col* 
lected by course title. Because California has engaged in a major effort to 
uj^rade its state-devdoped curriculiun frameworks, it is particularly 
important that new indicators measure the extent to which the content of 
those frameworks is reflected in school- and dassroom-level curricula. 

Georgia is currently developing a more comprehensive state indicator 
system, including a new course cat^rization system. An approach like 
the one used in Uie SRA study could help to answer a practical qu^on 
that state officials have asked: Can Georgia use a single designation for a 
course such as algebra I, or will it need multiple designations to distin- 
guish among very different levels and content? The G^rgia system of- 
fers three different diplomas— general, college preparatory, and voca- 
tional-^ach with di^^ent coursework requirements, and this affords an 
additional basis for measuring curricular differentiation.*' 



THE STUDY 
SAMPLE AND 
ITS 

LIMITATIONS 



^Both Califomia and Gcoi^ also increased course requirements for high school 
graduation in the t980». Califomia requires 22 Carnegie units, indudii^ 13 in particiJar 
courses: three years of English, two of mathematics, two of sdeiKe, three of social studies 
(including uA and wortd history and culture, economics, geography, and U.S. 
government), one of fine arts w foreign language, and two of phy^cal education. Th^ 
requii^menfis became effective with the gr»Iuatifig dass of 1987. 

Georgia requiim 21 units, of which 13 are specified by the state: four years of English, 
two of mathematics, two of sdence, three of social studies, one of physical education, and 
one of computer technok^, fine arts, or vocational education* The course requirements 
becante effective with the dass of 1988. 
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We used seven high schools as data sources for our validation study.'* 
These schools do not in any way constitute a representative sample of 
hi^ schools in California or Geoi^gia. Ourstiategy of combining surv^ 
with detailed bmchmark data required us to limit our sample to a small 
number of schools; and several of those we contacted refused to partici- 
pate.^ Nevertheless, the participating schools do vary in location, size, 
ethnic composition, and extent of ciuxicular differentiation.^ The sample 
is described in greater d^ail in the Appendix 

Although the sample is small and not repres^tative, it enabled us to de* 
velop a unique database for designing indicators that overcome the 
shortcomings of cum^tt measures^ The data enable coursework to be 
assessed in grater depth than is possible with the type of ^tistics now 
used in most state indicator systems; they provide a grrater span of 
variation than is possible in mc^t ethn(^;raphic studies; and they allow 
for the evaluation of a variety of statistical measures that are often used 
in large national studies but rarely judged as to their validity. 



DATA The five types of data collected in each school are summarized in Fig. 3.1. 

COLLECTION The teacher and student surveys are d^cribed below, and the remaining 
PROCEDURES data are reviewed in Section 5. 

Teacher Surveys We assumed that these smveys would eventually be administered as 

part of the process of routine state data collection; therefore they were 
designed to take teachers only about 30 minutes to complete. In every 
school, all of the teachers who taught any mathematics or social studies 
course in the 1987-88 academic year were surveyed. They were first 
asked questions about their educational background (e.g., number of 
mathematics or sodal studies courses taken, amount of subjects-matter in- 
service over the past three years) and experience. They were then asked 



^Ftve of the schools included grades 9-12; two of the Ceoi^ta schools contained only 
grades 10-12. For the Utter two sctmis, iiinlh-gnide nuthematics and sodal studies 
tead^ were surveyed at ^ junior hi^ sclux)ls that served as feeder sdtools. 

^Despite the problems that we experienced in gaining access to high schools, we 
decided not to reduce the scope of wt data collection (e.g., by eliminatii^ the transcript 
analysis or limiting the number of students surveyed). Had we done that access would not 
have been a probtem, but tte quality of our validation effort wrnild have been severely 
comprontised. We also felt that since state governments would be the agencies mc^ likely 
to field-test our indicators in the future, their authority to mandate such data coHection 
wc^d mean that the iiulicators would eventually be field^-tested on an entire population or 
a representative sample of high sdtools. 

^rve of the seven schools have a majority Anglo enroOmenl (55 to 65 percent); one 
has a majorihr of Hispanics; and the other has an enrollment almost equally divided among 
Angles, BlacKS, and Hispanics. The proportion of students going on to attend fi^r-year 
colleges ranges from 7 to 28 percent across the seven schools* lluee schools have a 
minima] amount of formal differentiation among levels of the curriculunv with only 
regular and honors classes offered; three are highly differentiated, offering up to four 
different levels; and one is moderately differentiated, with remedial regular, and honors 
classes in some subjects, but only regular and honors classes in others. 
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Fig. 3.1— SRA date sowem 



to give a period-by-period description of the classes they taught 
(including those outside mathematics and social studi^)^ to provide 
some information about teacher assignment patterns, and to indicate 
whether and in what ways any of thrae courses may have been affected 
by recent changes in state graduation requirements or other state poli- 
cies* 

Those teachers who taught any of the five courses being considered in 
the SRA were then asked to complete a separate survey (still included in 
the 3(Vminute time limit) for each section of a course that they taught in a 
significantly different way/ Teachers were asked about textbooks and 



^Respondents who taught multiple sections of the sanie course to students at the same 
ability level using the same instructional strat^tes were asked to complete only one form. 
Therefore/ the number of sections listed in the Appendbc is lower than the total number 
offered acrc^ the seven schools. 

If teachers taught more tjian two sections with significanlly different content and 
instructional approaches, they needed at>out 45 minutes to complete the survey. 
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other materials; topic coverage;^ the number of periods devoted to each 
topic; and whether the topic was tai^t as new content reviewed and 
extended, reviewed only, assumed as prerequisite knowledge, or not 
taught and not assumed as student knowledge (ess^tially the SIMS-IEA 
strategy for ascertaining depth of coverage)* Respondents were also 
asked about their instructional strat^es (an adaptation of the NAEP, 
lEA, and NELS items),^ their goals for the course, the types of assign- 
ments and examinations they gave, their distribution of grades, student 
preparation, and level of student performance, given that preparation. 

Student Surveys Our student surveys were designed as questicmnaires that states could 

administer in conjunction with their standardized achievement tests. 
Consequendy, these surveys were even shorter than those administered 
to teachers: Siuveys for tenth graders required approximately 10 min- 
utes, those for twelfth graders, 15 to 20 minutes. They were adminis- 
tered to all tenth and twelfth graders in attendance on a particular day. 
The surveys included items about each student's background and future 
educational plans, as well as questions that repeated the instructional 
strategy items ask^ of teachers^ In this way, the students could be 
linked to individual teachers, and the level of agreement between the 
two data source could be evaluated. 

Because this report is deigned as a policy guide, SRA project results are 
used here only as example to illustrate the range of couisework indica- 
tors that states might consider adopting and the relative merits of differ- 
ent strategies for collecting data on those indicators.^^ 

The goal of the SRA project was to aid in the development, rather than 
the production, of coursework indicators. Therefore, sample size and 
statistical representativeness were less important than capturing the 
range of variability in coutsework conditions and assessing the accuracy 



^The mathematics topics included in ihs survey (15 for algebra 11 and 23 for the other 
mathematics courses) are similar to those used in the SIM5-IEA study and the Center for 
Research on Evaluation, Standards, and Student Testing (CRE5ST) Instructional 
Assessment Project. Bfteen U.S. history and government topics were selected for each 
course; tl^se Liduded historical events, political institutions, and concepts (e.g., the 
potential conflict between liberty and equality). We based our choice on curriculum 
frameworics such a5 those in California and consultations with historians and political 
scientists. 

^Mathematics tochers were given questions about 14 different instructional strategies 
(e.g., tl^ review of homework pr^tems in class, small group work^ the use of calculators 
and computers) and the frequency with which they were used. History and government 
teachers were given questions about 12 instructional strategies (e.g., Iccturit^ to the dass^ 
student presentations, reading primary materials) aiui the frequency of their use. 

complete summary of study findings and a discussion of the technical issues 
involved in designing coursework indicators are included in The Design of hnpraved 
Coutsework Indicators: First Steps, available from the Center for Research on Evaluation, 
Standards, and Student Testing, UCLA Graduate School of Education, Los Alleles, 
California. 
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HOW THE SRA 
FINDINGS ARE 
USED IN THIS 
REPORT 



with whidi difiexBnt measiii^ described We collected 

more data than woiild be necessary fc^ a routine indicator system and 
traded large sample size for depth of information. As a xesult, issues of 
^tistical significance were of less importance in reporting findings than 
simulatUig the reasoning cme would apply in using indicatOT data to 
evaluate student onirsework. Measures of statistical significance are not 
reported in the examples discussed her^ althoiig^ for some analyses, 
they w^ used heuristically. The project findings should not be inter- 
preted as re|»esentative of the effects of coursework reforms in 
California or Geoxgia^ nor do the examples illustrate all of the informa* 
tion that coursework indicators can convey. 

The next section depicts sample coursew<^ indioitors developed in the 
SRA and describes ^ kinds of information that each measure generates. 
Section 5 uses study findings as a basis for asses^g alternative data 
collection strategies. 



4. SAMPLE COURSEWORK INDICATORS 



The basic elemente of student coursework considered here are content or topic coverage^ 
instructional strategic, curricular objectives, and teacher qualifications. Schools' perfciinance on 
each of these dimensions can be gauged in a variety of ways. This section presents indicators to 
measure each element and illustrate how they can be used separately and tc^ther to document 
what schools are teaching. 



TOPIC The niunber of periods teachers devote to a topic and the way they treat 

COVERAGE that topic in a particular section constitute a workable indicator that 

states might use to measiu^ coursework patterns and trends. Six alter- 
native topic coverage indicators and their major shortcomings are listed 
below: 

• Course offerings and enrollment based on gCTieric Htl^. These 
data do not indicate the variability in material covered in 
courses with the same title (see Section 1). 

• Textboc^ use. This measure is imprecise because it does not 
tell which parts of the text are actually covered in a particu- 
lar class or how teachers may supplement textbooks with 
additional material. Breadth and depth of coverage can be 
measured only through a time<onsuming process of map- 
ping textbook content with what is taught in the classroom. 

• Textbook plus chapters covered. This measure reduces the 
problem of inference about coverage based on knowing 
only which textbook is used, but some mapping is still re- 
quired. 

• Sifllabus or syllabus plus class assignments, exercises, and exami- 
nations. This measure, while more valid than the previous 
three, still requires considerable effort to ensiu-e that it is in- 
terpreted correctly. Inclusion on a syllabus provides no in- 
formation about how a topic was treated from the teacher's 
perspective;^ assignment and examination analysis depends 
on the comprehensiveness and representativeness of the ma- 
terial provided, and considerable effort may also be required 
to derive measures of topic coverage and treatment (le., the 



^The syilabtis as a measure of topic coverage and treatment within individual 
classroonos may become an even less valid indicator as schools and districts seek greater 
standardization of their courses. For example, in several schools in the SRA sample^ 
teachers used a department-ivide syllabus that was common to all sections of a course 
taught within that school. However^ our survey data tmlicate that despite this common 
syllabus, topic coverage varied across sections of the same course. One urban school in our 
sample used district-wide syllabi, and variation again occurred across sections* 
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breadth and depth of the curriculum as it is actually pre- 
sented in the d^room). 

• Opportunity-t(hleam (OTLK « measured &y test items. Under 
this approach, used in the SIMS study^ teachers are asked 
whether their students were taught the material necessary to 
answer specific test items; if they were not, why not ^d 
what proportion of students in the class could answer each 
test item. The OTL measure requires teachers to respond to 
items from a curriculum achievement test Itie SIMS re- 
searchers found that teachers may respond to the wrong 
characteristics of some items or may overlook the fact that an 
item taps multiple dimensions^ For example, mathematics 
items that deal with the a>ncept of similar triangles may be 
embedded in word problems or graphic^ representations, 
but teach^ may focus on the concept itself and not on 
whether students were taught to apply that concept in dif- 
ferent formats. 

• Ongoingr detailed classroom obsenmtions. This is the best way 
to make valid inferences about the curricultmi that students 
are actually receiving. However, classroom observation is 
not feasible for a statewide indicator system because it is 
time-consunung and labor-intensive. Standardization re- 
quires well-trained, experienced observers, and extensive 
observation is necessary to monitor topics as they are intro- 
duced and then later reviewed and expanded or extended. 

Topic coverage, in terms of time and treatment, is a middle-range indica- 
tor that balances fidelity to coursework coverage and emphasis with re- 
spondent burden and cost,^ Other ways of gaining in-depth descriptions 
of the curriculum have greater validity, but most means of obtaining 
such information on a large scale are either more crude (e.g, they repre- 
sent the intended rather than the implemented curriculimt) or more 
costly in terms of personnel, time, and resources. 

Topic coverage indicators can be used in several ways to provide data 
about coursework patterns. The discussion below describes some of 
these ways. Problems that may be encountered in operationalizing topic 
coverage indicators are also discu^ed* 



Topic Coverage 
Indicators and 
the Information 
They Provide 



^The SRA survey, administered in faU 19S8, asked teacher* about the number of 
periods they devoted to spedfic topics and the treatment Uwy gave them during the 
previous academic year. While this approadt may have sacrificed some accuracy, since it 
depends on teachers' ability to recall what they had taught over the precedti^ twelve 
months, it allowed them to d^cribe an entire course, based on their actual teaching rather 
than on their plans for upcoming classes. Data of this kind might be more reliable if they 
were collected in late May or early June for the oirrent academic yean 



ERIC 



35 



-22- 



Avenge topic coverage in U*S* history and government Figure 4.1 and 
Table 4.1 show the average number of periods that VS. history and gov- 
ernment teachers reported sp^idii^ on differ^t topics* These are ex- 
amples of the simplest uses of topic covera^ indicators^ but they convey 
some important informatioa For example. Fig. 4.1 that with the 

exception of westward and over^as expansion^ on which the Geor]^ 
schools spent an av^ge of two weeks more than the California schools, 
the sample schools in the two states averaged about the same amount of 
time on each topic However, as shown in Table 4.1, there was coi\sider- 
able variation among sections in the number of periods spent on a par- 
ticular topic. This variation onurred ncH just acrc^ the entire sample, 
but also within a given school. For example, in the suburban Georgia 
school, some teachers reported having spent only a week on the 
Constitution, while others spent two weeks; some spent less than a week 
each on the scope and limits of presidential power and on major social 
issues, while others spent more than two weeks on those topics. Given 
that U.S' government is a one-semester course, a difference of one week 
changes the a)urse emphasis ccnsiderably. In U.S. history, the amount 
of time spent on a particular topic differed by as much as three weeks 
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Table 4.1 



PERIODS SPENT ON U.S. GOVERNMENT TOPtCS 
IN HIGH SCHOOL HISTORY CLASSES 



Topic 



AvoraoeNo. 
of Periods 



Range 



as. Coratitution and Bfli of Rights 
Duoprocws 

PotOTtiai oonSiet betMfMH ltt)erty and equaltty 

Pedarsdist paper #51 

Conpws 

Role of biter^ ^oups 

Scope and Umfts of presktentia) power 

Court ^ttem 

Supreme Cowt decisions 
Political paities, ^ctiom, and vot^ 
State and local government 
Different forms of govonment out^ the 



BT 
3.5 

2.e 

0.5 

e.e 

2.4 
5.9 
B.2 
3.4 
6.9 
5.1 
4.0 



2-20 

1- 10 
0-7 
0-2 

2- 15 
0-5 

2--15 
2-20 
0-10 
2-15 
0-15 
0^20 



United Sta^ 
Major social tesues 
Rights and r^ponsibditi^ of citizens 



5.3 

5.0 



2-12 
1-12 



among classes within the same schooL The most significant variation oc- 
curred in analytical topics, such as the ideological origins of the 
American revolution^ and in topics h'om recent history, such as the dvil 
rights movement and the cold war. 

Uses of average topic coverage infomiation. Although the topics in the 
SRA instruments were selected to be neutral with regard to either curric- 
ular preferences or specific policies, the information gen^te^ from topic 
cov^ge indicators can be used to compare coursework patterns against 
a variety of standards. For example, it has been argued that the time a 
teacher spmds on a particular topic is largely detmnined by the amount 
of coverage the textbook devotes to that topic We performed a limited 
analysis to test the validity of that argument We examined four of the 
thirteen VS. history texts used by teachex^ in the SRA sample and calcu- 
lated the number of pages devoted to each topic as a proportion of the 
pages devoted to the fifteen topics listed on the survey. We then com- 
pared this with the number of periods teachers reported spending on 
each topic as a proportion of the total nimtber of reported periods. We 
found that the correlation between textbook and teacher coverage across 
all the history topics was 03; however, no consistent pattern emerged. 
For some topics, teadiers followed the relative emphasis of the textbook 
exactly; for others, they devoted considerably more or less attention than 
did the textbook. 
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Topic coverage might also indicate whether curricular patterns have 
changed with policy actions* For example, the data on California schools 
might be viewed as a baseline. The relative empha^ accorded different 
U.S. history topics might be expected to shift as schools implement the 
state curriculum framework that concentrates eleventh grade VS. history 
on the twentieth century. And in fact, teachers reported in their re- 
sponses to open-ended questions that they were be^nning to adapt to 
the new frameworks, for example, by moving quickly through earlier pe- 
riods in order to able to spend the entire second semester on the 
twentieth century. If data of the type collected in the SRA study were 
gathered regularly, this adaptation to new policy guidance could be ex- 
pected to appear in the quantitative indicators over a period of several 
years. 

The relative emphasis accorded different topics could also be used to 
assess the implemented curriculum according to different professional 
standards. For example. Table 4.1 presente a picture of U-S* government 
coursework that emphasize public law and formal institutions (e.g., the 
Constitution and the court system) and pays considerably less attention 
to political dynamics and policy outputs (e.g., the conflict l^tween lib- 
erty and equality, the role of interest groups, major Supreme Court deci- 
sions). Political scientists might argue that such a curricular approach 
portrays American government as more static than it actually is and that 
it minimizes the d^ree of conflict inherent in the political process. On 
the other hand, some educators might argue that since U^. government 
is for many students the last formal exposure to the fundamenUls of the 
American political system, a basic grounding in the legal structure of the 
system should be provided so that students can access the governmental 
services to which they are entitled and understand the terminolc^ used 
in televised news reports. Thus, although indicators should be as neutral 
as pc^sible, they should be designed to provide information that can be 
compared against a variety of standards. 

The need for more extensive social-studies indicator development 
Very little developmental work has been done on indicators of social 
studies coursework, in contrast to the amount of research that has been 
devoted to mathematics and science indicators. A few national studies, 
such as the NBLS, have included history topics in their teacher surveys, 
but the topics tend to \^ limited to very general categories of historical 
events. In contrast, mathematics indicator development has drawn on 
extensive work undertaken as part of achievement test, curriculum, and 
indicator development efforts. The choice of social studies topics for the 
SRA project was based on an examination of curriculum frameworks and 
textbooks and on a desire to include both broad and narrow topics in 
history, and in govenunent to include ones that spanned political institu- 
tions, policy processes, and more complicated analytical concepts. We 
believe that while our choice of topics has yielded valuable information. 
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more extensive developmental work like that already completed in 
mathematics and science should be imdertaken in sodal studies**' 

Proportionate coverage of mathematics topics. How much of an aca- 
demic subfecf s core content is covered in a typical dass?^ And does that 
proportion vary significantly across courses and sections? Coursework 
indicators can answer th^ questions, but they must be more sophisti- 
cated than just the average scores that are most commonly reported. 
Data from the SRA survey of teachers of algebra I and lower-level math- 
ematics illustrate how coursework indicators can be used for such a pur- 
pose. 

In developing indicators of coursework coverage in mathematics, the 
SRA project built on research begun during the SINfS study and ex- 
tended as part of CRESST's cooperation v^th the Math Diagnostic 
Testing Program (MDTP)^ in which a crc^-da^sification scheme for test 
items and curricular topics was designed and Held-tested (see Biu^tein et 
al., 1986; Burstein, Chen, and Kim, 1989). The 23 topics included on the 
SRA siuvey of algebra I and lower-level mathematics courses were se- 
lected from the MDTP Teacher Topic Coverage Questionnaire. They are 
representative of the content covered in algebra I and lower-level 
courses; some are quite general, and others are more detailed. Table 4.2 
lists the 23 topics, categorized into four broad groups.^ 

Figure 42 displays information about the distribution of topic coverage 
across sections in a more comprehensive way than is possible with sim- 
ple niunerical or graphical representations of the average number of pe- 



^Differences between the level of sophistication of mathematics and science indicators 
and that of sodal stwiies and English indicators may be partly due to the nature of the two 
disdplii^ mat!^matics and science consist of what is often perceived to be more 
hierarchically ordered knowledge, and they enjoy somewhat greater expert consensus 
about which content is central to the curriculum. However, the differing stages of indicator 
development are also tt^ result of significantly greater investment in science and 
mathematics curriculum and indicate research, laxgely the result of funding by the 
National Science Foundation. 

^^Core content^ can be defined in a number of different ways. For example, some 
might think of It as ctirricitlar expectations about what knowledge and skills instruction 
ought to provide and wouM Utus rely on prommiKements of gnnips such as the National 
Science Board and expert committees and on model curriculum franu?works for a 
definition. It might also be defirad through a review of texttx>oks, district scope and 
sequence guides, or past research on what is actually taught in a given subject 

We luve defiMd core matlwmatics content empiriduly rather than normatively, using 
past res^rch on teachers' topic coverage. However, we have also considered expert 
opinions about what should be induded in core content, as a basis for comparison. 

^This set of topics is not intended to be exhaustive for any course. Other topics that 
might have been sampled include ratio, proportion, and percent; exponents; proportional 
reasoning; basic properties of triangles; and the Pythagorean theorem. Including all of 
these would have broadened the coverage of high-level arithmetic and pre-algebra core 
material. Perhaps a better balance might be struck^ but we would be uncomfortable about 
expanding the list to ask teachers about more than 25 to 30 topics. While some adjustment 
may be needed in the l>alance among topics, we feel Utat the general strategy is sound. 
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REPRESENTATIVE MATHEMATICS TOPICS COVERED IN ALGEBRA I 
AND LOWER-LEVEL COURSES 



Afittmstfc 


P/e-aHoebfd^ 
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pdrcmtagds 








Oparatk>ns with radical and rational 


Solvtng quadratk: equations 




Qraphing equations and inequalities 


Opmtkms wWi polynomials 


Probabity 


Factc^ 


Statistics 


Solving Bnsar aquations 


Applications of pamlteiism and 


AppHcations of equations 


perpendioilarity 


Qeneratbig algebraic ec^ations from word 


Function oonospt, use of function 


problems 


notatton 


InecyiaBS^ 


&i^»Nng of functi^ 


Solutions of equations with absolute values 





*These topics are typt^y brlroduoad in pre-algebra courses, although brief introduc- 
tiof^ to thOTi are iBcely to occur ^ earfier OHirses. 

^These topics are virbjaBy imiversal tn standard first-year algebra courses. 

^These topics eitNr have no fixed place in the oirrtaikim (e.g., probabitity^ descriplive 
statistics) or are typi^rfly covered at ttie ^ of algebra I tM not emphasized until geome- 
tiy or algebra It. 

riods devoted to different topics (like those used to illustrate the 5RA 
social studies data). The plote in Bg* 4*2 convey considerably more in- 
formation about the distribution of topic coverage than do means and 
ranges* The line across the middle of each ''box'' represents the median; 
the lower and upper boundaries of the box equal the twenty-fifth and 
seventy-fifth percentile; the ^whiskers'" depict the tenth and ninetieth 
percentiles; and the dots represent outliers beyond the tenth and nineti- 
eth pen:rentiles. The lefi-hand side of Fig* 4.2 combines the arithmetic 
and pre-algebra topics, and the right-hand side combines the algebra and 
enriched topics. 

This example illustrates quite vividly that while students in a typical al- 
gebra I section were provided more exjxjsure to algebra and enriched 
topics than their counterparts in either pre-algebra or general mathemat- 
ics classes, the content coverage varied enormously across sections of 
courses with the same title. In the typical algebra I section, teachers re- 
ported covering 80 percent of the algebra and enriched topics, compared 
with 60 percent in the typical pre-algebra class and 27 percent in general 
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mathematics. However, the median scores mask considerable diversity 
within ^ch course; some pre-algebra and general mathematics classes 
covered as much alg?bra and enriched topics as some algebra I classes, 
although dte depth of coverage might have differed. Excluding the ex- 
treme values, the proportion of topics covered ranged from 44 to 93 per- 
cent in algebra I sections, and from 33 to 80 percent in pre-algebra. 
Teacheis in a quarter of the pre-algebra sections reported coveiliig at 
least as high a proportion of algebra and enriched topics as did the bot- 
tom half of the algebra I classes. The range of topic coverage was even 
greater across general mathematics sections, with the proportion varying 
from 0 to 67 percent. 

The same finding holds for arithmetic and pre-algebra topics. 
Particularly striking is the amount of variation within algebra I: 
Although 44 percent of the arithmetic and pre-algebra topics were taught 
to the typical algebra I class as new or reviewed/extended material, the 
proportion of lower-level topics covered in algebra I ranged from 13 to 
8S percent. 

Material of the type represented by this example is more complicated 
and difficult to report than most indicator data— esp^nally to diverse, 
nontechnical audiences- Yet it is precisely the kind of information 
needed by policymakers and educators who are concerned about curric- 
ular rigor and equitable learning opportimides. Such topic coverage 
indicators can provide different levels of detail for different audiences. 



... this example 
illustrates quite 
vividly that 
while students in 
a typical algebra I 
section were 
provided more 
exposure to 
algebra and 
enriched topics 
than their 
counterparts in 
either pre*algebra 
or general 
mathematics 
classes, the 
content coverage 
varied 
enormously 
across sections of 
courses with the 
same title. 
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... topic coverage 
indicators can 
provide different 
levels of detail for 
different 
audiences^ 



For example, the most important and also the mc^t general, condiision 
to be drawn from the data on topic coverage in mathematics is that be- 
cause course titles mask considerable variability, some algebra I students 
are receiving no more exiK)sure to algebra content than students enrolled 
in some general mathematics and pre-algebra classes* This is probably as 
much information as jMlicymakers and top*level administrators need or 
want the other hand, state and local curriculum experts need more 
detail about the configurations of topics covered in different classes and 
the amount of attention and type of treatment given them. Not all indi- 
cators can serve multiple pur^Kises, but wherever possible, system de- 
signers need to think about how to use the same indicator for reporting 
different types of information at different levels of detail to minimize the 
data collection burden. 



INSTRUCTIONAL 
ACnvmES AND 
CLASS 

ASSIGNMENTS 



A comprehensive picture of a curriculum requires information not only 
about the content covered, but also about the instructional strategies em- 
ployed to convey that content Key elements include the manner in 
which content is sequenced and the mode in which teachers and text* 
books present it to student For example^ research has shown that if 
students are to leam to use science aiKi mathematics as tools for dealing 
with real-world situations, they must be exposed to actual problems and 
not just routine exercises, and they need to leam how to apply a variety 
of organized approaches to problem solving* Thus, curriculum indica- 
tors ''should provide information about the degree to which students ex- 
perience the curriculum in a problem-solving mode rather than as exer- 
cises that are relatively straightforward translatiorts of procedures'" 
(Oakes and Carey, 1989: 110). However, such information (e.g., the 
number of problems presented in a class that are not direct applications 
of the textbook) is often too difficult or too burdensome to obtain in rou- 
tine data collection. 



Consequently, state indicator systems will have to rely on pn^xy mea- 
sures that sacrifice some validity and comprehensiveness for the sake of 
feasibility* Information about the mode in which content is delivered to 
students can be obtained, for example, by asking about the frequency 
with which teachers use different instructional strategies. This approach 
has been used in several national and international studies, including 
NELS, NAEP, the 1985 National Survey of Science and Mathematics 
Education (NSSME), and the SIMS, and it is the approach that was used 
in the SRA project. Asking teachers about instructional activities and 
class assigxmients is a cost-effident compromise between the status quo, 
where states have little or no information about how content is delivered 
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to students, and a detailed (and cc^tly) examination of instructional 
practices* 

We used two criteria in selecting activities and assignments to include on 
the SRA survey. First, we wanted a balance between commonly used 
instructional techniques (e.g., lecturing, having students read the text- 
book in class) and activitira associated with reform projMSsals to improve 
students' critical thinking skills (e.g., the use of hands-on materials or 
having students work in small groups, write research papers, and con- 
duct special projects). Second, since one purpose of the SRA project was 
to assess altanative data collection strat^es, we wanted to obtain in- 
formation about classroom activities from both teachers and students. 
Consequently, we selected a set of activities and assignments and a 
metric for reporting their frequency that we believed could be 
imderstood by both teachers and students.^ Where feasible, we included 
itenns horn the major national studies. 

Information about instructional strategies can be used in a variety of 
wa)rs to portray the curriculum that different types of students are re- 
ceiving. Figures 43 and 4.4 illustrate variation in activities and assign- 
ments by class ability level for homogeneously grouped U.S. history and 
government sections.^ 

The figures show significant contrasts across classes of different ability 
levels, but also several striking similarities. First, teachers lecture almost 
daily in the majority of classes at all levels. Second, the proportion of 
sections engaging in class discussion almost every day is the same for the 
low'-ability classes as for the high-ability classes. This finding clearly il- 
lustrates the limitation of using numbers of activities and assignments as 
a measure. It is likely that the content of class discussions differs across 
the two levels, as do teachers' goals and expectations in using this in- 
structional technique. However, such differences cannot be ascertained 
by measuring only frequency of activities. 



Using Activities 
and Assignments 
to Characterize 
Classes of 
Different AbUity 
Levels 



^Section 5 assesses the relative mentn of collecting coursework data from students and 
from teachers. 

^Thirty-five of the 66 history and government sections in the SRA sample were, by 
teacher report, homogeneously grouped, with al) students having the same ability level 
(either low, average, or high). The remaining 29 sections were reported by teachers to be 
heterogeneous, with a mixture of two or more ability levels. The profile of the mixed 
classes is most like that of the average-ability classes for all activities except lecturing and 
seatwork; the proportion of sections engaging in those activities almost every day is doser 
to that of the low-ability classes. 
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Fig. 4J--4¥squaiiey of MtivttiM^ 



Differences across levels for the remaining activities and assignments are 
consistent with the research on curricuhim stratification (see Gamoran 
and Berends, 1987). For example, considerably fewer high- than low- 
ability classes engage in seatwork alm<^t every day; but more than twice 
as xx^y high- and average^ility classes require student pxesentations 
monthly or more often. Reading primary materials and writing research 
papers are class assignments associated with teaching students problem- 
solving and critical-thinking skills* In the SRA sample, the high-ability 
classes are three times as likely as the low-ability ones to write research 
papers and twice as likely to read primary materials on a regular basis. 



The Effectiveness 
of Different 
Measures for 
Characterizing 
Class AbiUty 
Levels 



Evidence from the SRA study suggests that particular measures niay 
characterize courses more validly in some subject areas than in others. 
For example, although instructional strategies were found to distinguish 
among ability levels in social studies^ activities and assignments did not 
provide similar information for mathematics courses. The mathematics 
activities that occurred most frequently in the SRA sample — explaining a 
lesson to the entire class, reviewing homework problems, teachers work- 
ing problems at the board, and students working on problems alone— 
were similar across courses and ability levels. The activities and assign- 
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ments that occurred least often— the use of computers, students making 
oral reports, special projects, and mathematics labs — were also similar 
acrt^ mathematics course of all levds. Topic coverage and tr^tment 
were more effective for distinguishing among class ability levels than 
instructional strategies. Nevertheless, the activity profile of mathematics 
classes that emerges in the SRA sample provides other important infor- 
mation— e.g., that teachers are relying primarily on a traditional set of 
activities and assignments which differ from the reform proposals of 
professional mathematics groups. 




Lowabffity Average abDity Hi^abiBty 



Rg. 4*4~Froitueney of asstgnments, by dass abfltty 



On the other hand, whereas topic coverage differentiated among ability 
levels in the mathematics classes, it did not do so in social studies. The 
number of periods that high-ability classes spent on a particular topic in 
U.S. history or government was essentially the same as the time spent by 
average- and low-ability classes- This finding from the teacher surveys is 
consistent with schooMevel interviews that characterized the intended 
difference among classes for different ability levels to be the depth and 
manner in which content is presented, not the breadth of coverage, 
which is supposed to be similar for all classes. The varying effectiveness 
of different types of measures in d^cribing curricular stratification ap- 
pears to be valid, given the nature of mathematics and social studies. 
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no generic set 
of statistics 
should be 
assumed to apply 
equally to all 
courses. 



One would expect higher levels of knowledge accumulation to character- 
ize more advanced classes in mathematics, while deeper analytical rich- 
ness is the distinguishing factor in social studies* 

While data on both content and instructional strategies should be col- 
lected on all subjects, different measures provide more information about 
aspects of some courses than others. For example, in social studies, ac- 
tivity and assignment data are key to understanding the nature of in- 
struction received by students of different ability levels. The same data 
do not provide such information in mathematics, but they are important 
for monitoring changes in instructional strategies and teachers' re- 
sponses to reform proposals. Those who report indicator data need to be 
sensitive to the different types of information that indicators can provide; 
no generic set of statistics should be assumed to apply equally to the 
same aspects of all courses. 



COURSE 
OBJECTIVES 



In addition to topic coverage and instructional activiti^, coursework also 
includes the goals that teachers pursue as they present course content to 
students and use various instructional strategies* The relative emphasis 
teachers give to different objectives reveals something about their expec- 
tations for a particular course, and their choice of objectives is likely to 
influence how they configure topics and instructional activities within 
that course. However, teachers' reports of their course objectives reflect 
intended behavior and are likely to be less reliable than reports of actual 
behavior, such as topic coverage and instructional activities. For that 
reason, course objectives should receive less emphasis when decisions 
must be made about which data can realistically be collected from a 
teacher survey. 

Although coursework objectives can be a lead indicator of the direction 
in which coursework and teaching in a particular subject area may be 
heading, reports of teachers' course objectives should be interpreted cau- 
tiously. 

Table 4.3 lists 10 teacher objectives for a high school mathematics class 
and compares the proportions of teachers in the SRA sample reporting a 
major emphasis on each with the pro|X)rtions reported in the 1981-82 
SIMS sample. The SRA teachers, who were surveyed seven years after 
the SIMS teachers, appeared to have accepted more of the language of 
educational reform. The overall emphasis on objectives that appear reg- 
ularly in the literature on refomung mathematics teaching (e.g., develop- 
ing a systematic approach to problem-solving, understanding the logical 
structure of mathematics) was considerably higher for the SRA group 
than the SIMS groups. Conversely, emphasis on computational speed 
and acciu'acy has decreased. This pattern is consistent with reforms that 
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Table 4.3 



PERCENTAGE OF TEACHERS INDICATING A MAJOR EMPHASIS 
ON A SPECIFIC OBJECTIVE, BY COURSE TYPE 





General Math 


Pre-algebra 


Algebra 1 
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1 Mean 
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63 


61 


in everyday life 


68 


47 


54 






Developing an attitude of 














42 


76 


61 


60 


39 


Becoming interested in 












mathematics 


49 


71 


61 


60 


45 


Uncterstancflng the importance 












of mathematics in basic and 












applied scierK:e8 


42 


59 


46 


49 


20 


Performing confutations with 












speed and accuracy 


55 


47 


43 


48 


58 


Understanding the nature of 












proof 


12 


6 


25 


14 


12 



^n^se numbers represent aO teachers in tfte SRA sample who taught a particular course, if 
teachers taught mtiltiple sections of the same course, they were counted only once, but ff they 
taught two or more cOffarent ocHirses, ttoir response for eadi was counted separately. 

^Eighth grade teachers' reports of placing relatively more emphasis on a spedfic objective 
from data ooDected as part of the SIMS. 

^is objective was not ir^duded on the SIMS teacher survey. 

have been advocated since the early 1980s, although it may be exagger- 
ated by the prevalence of that reform ethc^ and perhaps by the some- 
what different mixttire of student populations in the two studies (the 
SIMS teachers were teaching middle school students, whereas the SRA 
sample were teaching high school students). We recommend caution in 
interpreting the changes in teachers' stated objectives because these same 
teachers report little use of instructional strategies designed to promote 
such goals. For example, although a high proportion of teachers re- 
ported a major emphasis on developing an attitude of inquiry and un- 
derstanding the importance of mathematics in the basic and applied 
sciences, activities consistent with thc^ goals (e.g., having students 
make oral rep>ortS/ do special projects, or use hands-on materials) were 
infrequently used. The most frequently employed instructional strate- 
gies were more consistent with traditional objectives. 
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... report of 
course obfectives 
should he in- 
terpretedinUte 
context of Uie 
tefonts being 
advocated at 0w 
time data are 
coUectedfOnd 
they should be 
compared witit 
teachers' reports 
of topic coverage 
and instructional 
activities* 



TEACHER 
QUAUnCATIONS 
AND TEACHING 
ASSIGNMENTS 



Table 43 also shows that algebra I and pre-algebra courses place greater 
importance than general mathematics courses on higher-level^ critical- 
thinking objective^ (e.g., a systematic approach to problem-solving, un- 
der^anding the lc^<^ structure of mathematics) and objectives dealing 
v^th mathematical ri^r and und^tanding (e.g., und^tanding con- 
cepts and knowing foOs, principle, and algorithms). In contrast, the ma- 
jor emphasis in general math^natics course is on fundamental arith- 
metic ddlls (Le., performing computations with speed and accuracy) and 
the iix^x)rtance of mathematics in everyday living. Even the specific pat- 
tern of relative emphads in pre-algebra is consistent with this course's 
dual role as a bridge to formal algebraic and geometric methods and as a 
m^ns of solidifying arithmetic skills and motivating continuing interest 
in the study of mathematics. 

As Table 43 illustrates, information on the objectives teachers emphasize 
does help in differentiating among course levels, and it suggests lere 
particular subject-matter fields may be heading in their instructional con- 
tent and practice. Howev^, reports of course objectives should be in* 
terpreted in the context of the reforms being advocated at the time data 
are collected, and they should be compared with teachers' reports of 
topic coverage and instructional activities. 

A valid picture of student coursework requires information not only 
about what is taught and how it is taught, but also about ^ho teaches it. 
As concern about the quality of the teaching force has grown over the 
past decade, the federal government and individual states have moved 
to collect data that would enable them to assess the match between 
teachers' qualifications and their teaching as^gnments. The annual 
Schools and Staffing Survey (SASS) sponsored by the U.& Department of 
Education asks a nationally representative sample of teachers about their 
educational backgrounds (BA major and minor), areas of certification, 
and years of experience, and also asks for period-by-period descriptions 
of their classes. Although individual states typically do not collect such 
detailed information on teachers, a growing number are collecting 
teacher assignment data that can be linked to state databases on teacher 
certification status. Twenty-three states can now report the prr portion of 
teachers spending a majority of their time teaching a particular subject 
who are certified in that subject (Blank and Schilder, 1989). 

However, the match between teaching assignment and certification sta- 
tus provides only limited information about teachers' qualifications. 
Darling-Hammond and Hudson (1989) point out that "the relationship of 
certification to particular kinds of preparation is imcertain. An uncerti- 
fied teacher may lack pedagogical courses, subject-area preparation, or 
student teaching experience, each of which has different implications for 
a teacher's knowledge and experience base-" They recommend that indi- 
cators of teachers' subject-matter knowledge will require data on the 
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type of £actois induded in the national SASS survey, le., o>llege major 
and minor and the number of courses taken in particular subjects. 



Different conclusions about the match between teacher qualifications and 
teaching assignment can be readied when academic major is used as the 
standard instead of certificatioa ¥m examine, of the 310 mathematics 
sections in the SRA sample,^ 83 perc^t were tai^t by teachexs certified 
in madiematics, but only 58 percent of the teachers had majored in math- 
ematics or mathematics education. 

Hgure 43 presents a graphic comparison of teacher qualifications, using 
the two different standards. It also compares teacher qualifications 



Teachers w^BA in malhamaScs Q OwSSsdIteactMNS 




tfigonoimfry. 



1^ 4,9-^3iialifieitiora of ma^^ 



^Because of the my data were collected on teacher qualifications, all of the teachers in 
the seven SRA schools who taught at least one mathematics or sodat studies course were 
asked about their qualifications. Thus we were able to collect data on 206 other mathemat- 
ics sections, in addition to the 104 that were the primary focus of 0^ study* 
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across different levels of mathematics courses. As might be expected, the 
classes above algebra I have the highest proportion of teachers who have 
a BA major in mathematics or mathematics education. However, there is 
little difference in the proportion of such teachers across the lower three 
levels. 

Years of teaching experience is another criterion by which teacher quali- 
fications are traditionally assessed* Research has shown this measiure to 
be correlated with teacher effectiveness. Although thrae is some evi- 
dence that after too many years, teachers may experience 'Immout;,'' or a 
drop in performance, the consensus is that at least to some point, 
^exprience improves teachers' performance'' (Darling-Hammond and 
Hudson, 1989: 75). Hgure 4.6 shows SRA teachers' years of ^cperience 
acrcKs three categories of mathematics classes. Comparison of Figs* 45 
and 4.6 reveals that experience distinguishes among those who teach 
various levels of mathematics more than academic qualifications. The 
most striking finding is «he high proportion of novices teaching lower- 
level mathematics classes: 42 perc^t of the remedial, vocational and 
general mathematics sections are taught by people who have been teach- 



er 5-10 10-15 15-^ 30-35 3&»- 

Years ctf6)qwi9no» 

Fig. 4.6---IHstributton Of matlM^^ 
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ing for five years or less, as compared with 19 percent of the pre^algebra 
and algebra I sections and 13 percent of the classes above algebra I. In 
contrast, 65 percent of the classes above algebra I are taught by people 
who have been teaching for more than 15 years; 54 percent of the algebra 
I and pre-algebra classes and 39 percent of the lower-level mathematics 
classes are taught by these senior teachers. 

Data on teachers have typically been used to describe the current supply 
of teachers and as a factor in forecasting future demand. However, as 
these examples illustrate, data on teacher qualifications, experience, and 
assignments can also be used to develop a pictiue of student coiusework 
in a state. In addition to comparing the distribution of teacher character- 
istics across difiierent course levels within one subject area, these data can 
be lised to compare qualifications and experience across different subject 
areas and acrt^ differ^t types of schools and commtmities. Such in- 
formation might, for example, be used by policymakers and educators in 
deciding what kinds of teacher incentives are likely to be most effective 
for staff development efforts, or in assessing the feasibility of new cur- 
ricular directions. 



The indicators discussed in this section illustrate different ways to depict 
the breadth and depth of student coursework, the distribution of key 
curricular elements across different course levels, the congruence be- 
tween the intended and implemented curricula, and teacher characteris- 
tics. But these examples are by no means exhaustive of the ways indica- 
tor data can be used, nor will they be equally useful to aU states and local 
districts or to all types of policymakers and practitioners. One critical 
task in the indicator design process is that of deciding what kinds of in- 
formation are most needed by those in different roles, and which mea- 
siu^ can best be reported in multiple ways to meet these differing in- 
formation needs. 

We return to the challenge of reporting indicator data in Section 6, but 
first we examine some of the strategies that states can use for collecting 
data on student coursework. 
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5. DATA COLLECTION STRATEGIES 



The tradeoff between ensuring high technical quality and minimizing cc^t and respondent 
burden becomes mc^t evident when indica^ systan designers have to dedde which data collec- 
tion strategies to use. States that are currently a>llectiing information for coursework indicators 
have selected the least expen^ve and least buidensCTie option, school-levd statistics categorized 
by conventional course titles. However, as the technical limitations of these data become more 
apparrat and policymakCTS d^nand better information, states will have to seek alternative data 
collection strategies, even if those strategies entail greater cost and respondent burden. Although 
such in-depti) methods as classroom ol^^ervation will never be feasible for routine indicator data 
collection, several of the approaches used in the SRA study represent reasonable alternatives. 
This section assesses th<^ approaches* 

STUDl^T One of the least o^tiy and least burdensome methods of collecting data 

SURVEYS VS» on student coursework is to survey stud^ts about what occurred in 

TEACHER their classes* Such surveys coukl be appended to the periodic standard- 

SURVEYS ized teste that most states administer. Although students would be 

asked to spend additional time, the Ic^istical problems would be mini- 
mal, since no new scheduling would be required, and the added coste 
would be maiginal. 

Students, however, may provide less reliable reports about course con- 
tent and method of instruction than teachers* Unlike teachers, students 
do not have lesson plans to refer to in estimating the time spent on dif- 
ferent topics, and their recall of content coverage may depend on the ex- 
tent to whidi they have mastered that content. Their recollection of class 
activities and assignments is likely to be more accurate (ie., they can be 
expected to remember how often they had to write research papers or 
whether the teacher lectured every day or several times a week). Student 
accounts of some activities, however, may be only partly correct becavise 
individuals can report on only what they have done, not on what the en- 
tire dass has done. For example, in a mathematics class, some students 
might work on problems at the board while others work in small groups 
or alone at their desks; respondents thus may be able to report on the 
frequency of only those acti^ties in which they participated most often. 
Only the teacher can provide a full accoimting of the entire range of dass 
activities. 

Surveying teachers, however, also has its drawbacks. Mc^t state indica- 
tor systems typically do not collect data directiy from teachers, or they 
collect only a nominal amount. A recurring complaint of teachers is that 
the paperwork burden imposed on them inhibits their ability to teach 
effectively. In addition, some teachers who perceive that their responses 
might be used to assess their performance or judge the appropriateness 
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of their teaching strategies may give socially desirable r^ponses rather 
than accurate reports of what actually occiured in their classrooms. 

In the SRA study, we sou^t to ass^ these tradeoffs by collecting 
counsework data bom both student and teacher surveys. We concluded 
that despite their lower cost and burden^ student surv^ are not reliable 
substitutes for questionnaire data obtained from teachers, even though 
there was a reasonable level of agreement between students and teachers 
on the frequency of classroom activities. Our reasons are threefold. 

First students are imable to answer a ntunber of questions that are criti- 
cal to understanding tl^ range of curricular offerings across a state — 
questions about teach^ qualiBcations^ the topics covered and whether 
ttiey were presented as new or reviewed material, and the ability level of 
the students in a given class. Students can really report only on class- 
room activities and assignments. And while sudi information is useful 
for depicting curricular depths it tells nothing about curricular breadth, 
teacher qualifications, or how these factois vary across courses and stu- 
dent ability levels. 

Second, our data suggest that for some course levels^ student-provided 
data may be very incomplete. For example, over 90 percent of the stu- 
dents in the SRA sample who took a basic mathematics class either did 
not report their teachers' names or did not complete the activity grid on 
their nuveys. In contrast, the nonresponse rate for algebra I students 
was only about 12 percent 

Third, there seems to be no consistent pattern of student disagreement 
with teachers on the frequency of activiti^ For e)cample, in US. history, 
43 p^cent of the students whose teachers reported small-group adiviti^ 
occurring once or twice a month disagreed, but as many reported that 
the activity happened more frequently as reported that it happened less 
often. Similarly, about half the algebra I students agreed with their 
teachers' estimates of the amount of homework assigned, but of those 
who disagreed, half reported it requiring more time, and halt le^. 
Where disagreements existed across nine algebra I activities, students 
estimated the frequency to be more than that reported by teachers for 
five activities and less for four of them. 

It is important to note, however, that there was considerable agreement 
between students and teachers on the most fi^uently occurring activi- 
ties. For example, 86 percent of the U.S. history students whose teachers 
reported lecturing every day agreed; 69 percent of the algebra I students 
whose teacheis reported explaining problems to the entire class almost 
every day agreed, and 71 percent agreed that teachers reviewed home- 
work problems almost daily. 
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Despite our strong belief that routine coursework indicator data can best 
be collected front teachers, we did detect that some teadier responses 
may be less reliable because of the social desirability pressure men- 
tioned above. Student descripticms of classxtx)m activities in U.S* history 
generally matched thc^ of their teachers, with two notable exceptions: 
lecturing and having students read from the textbook in class* Alx)ut 45 
percent of U.S* history teachers claimed to lectiare only once or twice a 
week/ while 78 percent of their students reported that the teacher lec- 
tured almost every day. Sttidents also reported substantially more read- 
ing from the text than their teachers. The majority of U*S. history teach- 
ers reported that they required students to read the textbook in dass no 
more than once or twice a months and a substantial proportion reported 
giving such assignment even less frequently. But 75 percent of their 
students reported reading the text in dass more often. While these were 
the only disoepandes that appear to stem from sodal desirability con- 
cerns, they do suggest a need to validate teacher Ksponses periodically 
with the type of benchmark data described later in this section. 

In sum, we condude that coursework data are best obtained through 
teacher surveys and that student surveys should be used to obtain in- 
forn\ation that will increase our understanding of schooling outcomes 
and student characteristics (e.g., family background, home support, aspi- 
rations, schooling preferences). Although student-teacher agreement is 
l\igh on some of the most common classroom activities, students' inabil- 
ity to provide infora:iation on key aspects of the curriculum and differ- 
ences in the completeness of data acrres course levels strongly suggest 
that the tradeoffs in data quality and completeness of teadier reports 
outweigh the added cost and teacher burden. Burden can be minimized 
by collecting coursework data on<^ every two or three years and by fo- 
cusing on a few critical courses in each subject area (e.g., data on ninth 
and tenth grade mathematics may be more important than data on 
courses offered in the last two years of high school)* Coursework data 
could also be collected on a cyde; for example, English and social studio 
could be collected one year, and mathematics and sdence the next. 

The financial cost of collecting coursework data from teacher surveys is 
negligible. We estimate that a stage's costs would amoiuit to less than 
one dollar per teacher and would probably be lower if the surveys were 
linked to existing data collection efforts and optical scanners were used 
for data entry. However, each school would need to devote about a half- 
day of staff time to ensure that siuveys were completed and collected, 
and individual teachers would need to spend between 30 minutes and 1 
hour on the surveys. As noted above, these burdens could be reduced 
substantially by collecting coursework data less often than student 
assessment data and by rotating the subject areas on which information 
is collected. 
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The major cost li& not in collecting or re{H)rting coursework data, but in 
indicator development. So much design work has already been done on 
mathematics indicators that states could simply adapt existing measures 
to their own information needs, but our res^jY:h suggests that in other 
subject areas, states would need to invest in an approximately two-year 
development effort Hiis process might involve expert anal^is of text- 
books, cuniculum frameworks, and professional standards, as well as in- 
depth teacher interviews; these aclivitira would have to be followed by 
the omvening of consensus panels of teachers and subject-matter special- 
ists to i^ch agreement on the major items to include on teacher stuveys. 
In one sense, then, collecting coursework data through teacher surveys is 
very cost-efficient and can be quite easily added to a state's existing data 
collection systent However, the data will not^ccurately measiu^ what is 
actually being taught in the state's classrooms unless the state is willing 
to invest in a thoughtful and well-executed design process. 



Interpreting coursework data from state indicator systems and national 
studies has traditionally been limited by the lack of forn\al analyses of 
their validity. Indicator designers have tended to concentrate m<»t of 
their energy on developing achievement-test items. Despite major ad- 
vances in the design of backgroimd and school process measures, studies 
have generally developed a few new items and then '1x)rrowed" others 
from earlier studies or from other states. Little effort has been made to 
validate these measures by comparing the ixUbrmation they generate 
with that obtained through alternative measures and data collection pro- 
cedures. For example, are teachers' reports of curricular goab or content 
coverage consistent with the material tested and the types of questions 
asked on their final exams? Are teachers' characterizations of the ability 
level of their classes consistent with the assignment policies described by 
school-level officials or the curricular paths of different students that are 
indicated by transcript analyses? 

Ensuring the validity of the inferences drawn from indicator data re- 
quires the collection of benchmark data, i.e., data that serve as standards 
or anchors against which the validity of the routine data can be judged.^ 
Benchmark data are more difficult and costly to collect than routine indi- 
cator data and can be thought of as a series of deeper probes that do not 
need to be collected as often or oit as laj^e a sample as conventional indi- 
cator data. 

Benchmai'V data can serve four purposes in a state indicator system: 



... collecting 
coursework data 
fhrou^ teaclter 
mtrveys can be 
quite easily added 
to a state's 
existing data 
collection system. 
However, the 
data toill not 
measure what is 
actually being 
taught unless the 
sfate invests in a 
well^executed 
design process. 

BENCHMARK 
DATA 



...ensuring the 
validity of the 
inferences draim 
from indicator 
data requires the 
collection of 
benchmark data 
that serve as 
standards. 



^Benckmrk data should not be confused with baseline data. Baseline constitutes the first 
point in a time scries, and baseline data are used to estimate change over some period, 
baseline comparisons usually use routine indicator data and do not require the special 
collection procedures of benchmark data. 
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• As of the indicator design process, l^nchmark data can 
inform the development and evaluation of new measures 
because the information thqr generate is much dc^r to the 
content and process of instruction than more routine data 
(e.g., in-depth interviews wiUi teachers or classroom obser- 
vations vs. teacher surveys), and they are less subject to ex* 
traneous influences that limit or compromise routine data 
(e.g., missing information due to respondents not completing 
surve3rs). 

• After an indicator system has been operating for some time, 
benchma^ data can be used to assess the extent to which 
measures have been corrupted by social desirability or pol- 
icy prresures (e.g., teachers may be reporting classroom be- 
haviors that are consistent with reform policies, while their 
examinations do not reflect the reforms). 

• Benchmark data can provide a context for interpreting 
trends or unexpected change in aggregate indicator data 
(e.g., through sdiool-level case studies). 

• Benchmark data can be used in spedai studies to comple- 
ment and enhance indicator syston data. These studies can 
explore areas of policy or practitioner concern that indicator 
data have signaled as problennatic (e.g., that curricular op- 
portuniti^ are becoming more unequal in certain types of 
schools). 

Benchmark data may take a variety of forms. In the SRA study, we col- 
lected such data by interviewing district and school-level staff, coding 
student transcripts, and reviewing teachei^' syllabi axul course materials. 
Other typ» of coursework benchmark data collection include regular 
classroom observations and alternative student assessments adminis- 
tered outside the higher-stakes conditions associated with standardized 
testing by states. 

School-Level To obtain benchmark data in this study, we interviewed the principal. 

Interviews and the head counselor, and the mathematics and social studies department 

Case Studies chairs at each school in the SRA sample. The district-level staff responsi- 

ble for supervising the high school curriculum were also interviewed- 
These interviews typically lasted about one hour and were often fol- 
lowed by additional telephone inquiries. Our purpc^ was to obtain in- 
formation on the types of students attending each school and whether 
the composition of the student body had changed recently; the levels of 
course offered and whether the curricular differentiation had the same 
meaning across academic departments; what criteria each school used in 
assigning studente to different courses and sections; how decisions about 
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teacher assignment were made; and how recent state policies may have 
a£tected each school's course offerings and instructional practices. We 
also asked the departm^ chairs to describe in some detail the major dif- 
feroiras among the five rourses selected for tfie SRA study in terms of 
level of difficulty, tyP^ of students craoUed^ topics covered, instruc- 
tional materials and strategies, course requirements, and grading prac- 
tices* 

The school-level interviews served several purposes. Fust, they allowed 
us to explore the tei^bility of collecting coursework data from sources 
other than t^diers in order to reduce the teachers' paperwori; burden. 
We found a reasonable level of agreement acrras the principal, head 
counselor, department chairs, and other teachm on how the curriculum 
is cn^anized in a gi v^ school Differences were most evid^t in reporte 
about the e^cts of various poli^ changes. The teach^ surveys and the 
intmrlew data were in general agreonent about how recent state and 
district policies had influenced curriculum, but administrators touied to 
perceive a greater impact than the teachers did and were more positive 
in their assessment of the effects- 

The SRA interview data suggest that respondents above the level of de- 
partment chair caimot report aixurately on how curricula vary across 
student ability levels within a particular department or a>urse* Higher- 
level respondents tend to underestimate the extent of variation, and even 
department chairs reported less variation across sections than analysis of 
the teacher surveys indicated. 

A second purpc^ of school-level interviews was to examine the formal 
policies that influence curricular stratification within schools. Policy- 
makers who are concerned about differential learning opportunities need 
information on student assigiunent policies and how they differ across 
schools. For example, what role do test scores, grades, teacher recom- 
mendations, and student and parent preferences play in assignment de- 
cisions? How difficult is it for students to change their course levels, and 
\mdei what <x>ndition$ do such shifts oow? We found that reliable in- 
formation on school policies can be obtained from either the principal or 
the head coimsdor, and a survey of such policies might omstitute the 
kind of special study that would enhance indicator data. 

Finally, schooMevd case studies can place indicator data in a richer and 
more valid context and thiis facilitate the interpretation of trends in ag- 
gregate data. For example, case studies could identify movements to- 
ward more heterogeneous grouping in schools and could indicate how 
teachers are adjusting to such changes in their content coverage and 
classroom activities. 
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Transcript Data At each school in the SRA sample, we examined the transcripts of stu- 
dents who were ninth graders in 1982 (1983 for Georgia), 1986, and 1988. 
These three dass years were selected because the students who gradu- 
ated in 1986 in California and 1^ in Georgia were the last to graduate 
before state-mandated increases in coiu^ requirements took e^ct; 
the dass of 1^ was one of tiie first classes under the new requirements, 
and we wanted to examine coursetaking by a dass that took U*S. history 
the prior year; finally, the dass of 1991 provided an opportimity to 
examine the previous year's coursetaking in lower-level mathematics 
and al^bra I. 

Each transcript was coded to indude student background (gender, eth- 
nidty, birthdate, grade-point average, standard!^ test scores, and 
niunber of absences).^ llie academic level of each course (in ntathemat^ 
ics, social studies, English, sdence, foreign language, vocational educa-^ 
tion, fine arts, and miscellaneous^ was categorized as either 

• Remedial— instruction aimed at remediating basic skill defi- 
dendes. 

• R£gular/Zwir— academic material prraented in a manner 
suitable for students who will end their formal schooling 
with high school, emphasizing exposure and basic compe- 
tendes. 

• Applialfvocatioml'— content foaxsed on students' possible vo- 
cational objective, emphasizing applications in the work 
setting. 

• Heterogeneous — material appropriate for students vnth a va- 
riety of abilities and educational objectives. 

• College preparatory — ^material that gives students academic 
skills and breadth of exposure suffident to prepare them for 
college-level work.^ 



^We attempted to collect data on students' sodoecc»H)mic status as well, but this effort 
did not produce consistently retiaUe information^ so tiie data were nc^ used in anjr of our 
analyses. We did obtain data on which students qualified for free or reduoed-price 
luncW. While this measure tends to provide an accurate count of the number of low- 
im^me students in rural high sclmls, it is not reliable for uxbm high sctools because a 
significant number of students who qualify do not sign up for the program because of 
emharrassm^t or for o^^mr reasons. We also attemf^d to obtain information on which 
students w^ living in ^nglenparent lunsseholds. This measure was reasonably reliable for 
six of the seven sample school, but tte scvCTdi school, as a matter of policy, entered only 
one parents name on the student reconi whetter the child lived with one or two parous. 

^ThQ miscellaneous category included physical eduction, driver's education, health 
and sex educatim, computer Hteracy (as c^pposed to computer fmgramming and computer 
sdence classes, which are included in mathematics), and ROTC 

^The distinction between the college preparatory and heteiogeneims levels is based 
more on the kinds of students in each dass than on cmtrse content Since heterogeneous 
classes include students who are college-bound* their content must meet tl^ standards for 
coDege-preparatory courses init also be appropriate for students of other ability levels and 
educational aspirations. The major diffei^ce between the two course categories is that 
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• Honors-college preparatory content, but enriched or accel- 
erated* 

• Advanced — material that prej^r^ students for advanced 
placement (AF) examinations. 

Courses were given level designations for two related reasons. First, this 
enabled us to diffoientiate amcmg sections of coiuses such as VS, history 
or English in which the same title may ma^ significant variation in con- 
tent and academic rigor within individual schools and also across institu- 
tions. Second^ some course titles have con^tent meanings within a par- 
ticular schcx>l but differ across schools. The level d^gnation clarifies the 
nature of such courses to thc^ out^de the school and helps in standard- 
izing course definitions. For example, one school in the SRA sample of- 
fered world history at the college prefwatory level and geography at the 
basic level The diffor^ce in subject-matter focus was less important 
within that school than the feet that the two courses were used to track 
students who all needed to meet a social studies requirement. In another 
school, however, a course with the title "world history^ was offered to 
students at both the basic and honors levels* Without the level designa- 
tion^ important differences in course content could go unrecorded in an 
indicator s^tem based on only standard course titl», particularly in the 
case of English, history, and social studies courses.^ 

Adding a level designation to a transcript coding scheme presented sev- 
eral difficulties in the SRA transcript analysis. First, the categories were 
defined to represent different levels of academic rigor in course content. 
However, content may be confcnmded with school placement poUdes if 
courses are categorized primarily by the ability levels of the students tak- 
ing them, rather than by their content. A second problem is that while 
maintaining consistency in coiuse categorizations within schools is fairly 
easy, keeping that consistency across schools is difficult. A t>asic course 
in one sdiool for example, may not have the same level of academic 
rigor as the same course in another school Although we foimd that the 
SRA scheme distinguished among coiuses better than did standard 
titles, in a few instances a a>llege pre]^ratory course in one school was 
closer in content to the basic course offered in another school than to the 
college preparatory course at that school. Part of the reason for this lack 



coU^e-pr^watory denotes courses whidi include only students identified as achieving at 
a level qiuUMng for college admission^ while heterogeneous dass^ include students with 
a range of abmty levels. 

*Foiir additional pieces of infbnnation were coded for each course: whether the 
course was intended for a special population such as handicapped or hnuted-English- 
proficiency students; wl^ me course was takai; tl\^ grade each student received; and 
whether the course was taken at the school under study or the credit was transferred from 
another school 

The coding of this information was based on in-depth interviews with school staff, a 
review of course handbooks and other materials^ and follow-up telephone inquiries, as 
needed for clarificatioa 



of equivalence is that some schools are more differentiated than others. 
For example, in a school with no honors or AP da^es, the coUe^ 
preparatory courses may have hi^er levels of academic content than 
they do in schools that have the additional course levels. 

Despite the challenges they present to indicator designers, transcript 
analyse illustrate two important us& of benchmaric data. First, they 
play an important role in the indicator design process. For example, in 
the SRA project transcript analysis was used to develop and test mea- 
sures of course levels. If these m^isures were found to be valid in coding 
transcripts, they could then be tested on other data collection instru- 
ments such as teachra surveys. One test of their validity is whetl^ the 
level of a course a ^udent takes in one year accurately predicts the level 
of the next year's course, afkOT other relevant foctors sucii as student gen- 
der and ethnicity, school attended, standardized test scores, and course 
grade are taken into consideration. Statistical analyses showed that ini- 
tial course level, al<mg with t^t scores, course grades, and school at- 
tended, were significant in explainii^ stud^t course levels from one 
year to the nexL^ The next step in the indicator development process is 
to refine the level categories and to include them on a teacher survey to 
determine whether they have the same miming for characterizing 
classes as they do for the transcript data. 

Trai^cript data can also be used in a spedal-studi^ component that en- 
hances routine indicator data. For example, teacher surveys allow char- 
act^izations of student composition only at the level of sections. If poli- 
cymakers and educates are interested in which students are taking 
which courses, transcript analyses can provide a much more complete 
picture of the learning opportunities afforded different students by using 
individual-level data. Transcript data can show the curricular paths 
through high school of vocational as compared with academic students, 
of minority students, and of boys as compared with girls. 

Transcript data can also augment cn^s-sectional survey and enrollment 
data by tracing a>ursetaking patterns over time by student cohorts. This 
is particularly important in a time of major changes in coursework poli- 
dra. In the SRA sample, we found ttiat eleventh graders in the pre- 
reform cohort generally had about the same total years of coiirsework in 
each subject area as thc^ in the post-reform cohort Only for sdence 
course in the California schools did the av^ge number of years taken 
increase by at least half a year (from an average of 1.5 to 2 years). 
English was typical of the other subjects: The average number of years 



^The analytical procedures used to tesi this aspect of the transcript coding schme's 
validity are described in the SRA prefect technical report. 



-47- 



taken by elevenUi graders in the Georgia schools remained at Z9 for both 
cohorisi; it remained at 3.1 for both cohorts in the California schools. 

However, some change did occur in the level at which courses were 
taken. Hgure 5.1 shows Ae trends in English coursetaking in grades 9 
throi^ 11 for the two colunts; similar trends emer^ in other subjects 
asweO. The ma^ change was a shift in the distribution of coursework 
in the California sctods. The proportixm of students who took all col- 
lege prqiaratory and heterogeneous classes incr^sed by one-quarter, to 
SSpocentofall^udents. This shift w^ counterbalanced by a decline in 
the proportion of students who took some remedial and basic classes. 
We conclude that in those schools, move studoits were given the oppor^ 
turdty to take more aradonirally ri^nous courses. The shift in the 
Georgia schoc^ presents a mi?^ picture. The proportion of students 
taking all honors and AP courses increased sUghtly, from 9 to 12 percent, 
but at the same time, the proportion of students eiuolled in all basic and 
remedial courses also increased, from 29 to 34 permtt 
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A simpler statistic that summarizes change in coursetakii^ over the re* 
form paiod is the shift in the proporticm of students who completed the 
elevenUi grade having tak^ no mathematics courses at or above the 
level of algd^ L This is an example of an indicator that is well under- 
stood by a variety of audiences and tiiiat carries significant information 
about changes in the acadfflnicrigctfc^ student couxsework. Thepropor- 
tion of stud^ts with this limited mathematics exposure declined in the 
California schools, from 33 to 25 percent, but in (he Georgia schools, it 
incrrased from 38 to 45 percent 

The cost of assembling and coding student transcripts prevents their use 
as a routine data source. Our experience suggests that about four profes- 
sional staff days per school axe required to draw a sample and develop a 
course dassifkation schen^ that is both valid for that particular school 
and comparable across schools. Assembling and reproducing the tran- 
scripts takes about three days of clerical staff time per school and coding 
takes from 12 to 15 minutes per transcript 

However, transcript analyses can be an important resource in designing 
new measures and oostHsffident way of conducting longitudinal stud- 
ies that mea^ire change over several student cohorts. With more and 
mOTe districts now (x>mputerizing transcripts, states may be able to make 
future traascri^t data collection less costly by encoura^g local districts 
to adopt software that stores tite data in a foirly uniform manner and in- 
cludes some information about course levels. 

Course Materials Data on course materials proved to be the most problematic type of 

benchmark data. We had originally hoped to collect sample assign- 
ments, as well as course syllabi and final examinations. However, a pre- 
test indicated that such an effort would be burdensome to teachers and 
would be difficult to interpret validly (e.g., the data wouM not indicate 
whether the oiUected assigmx^t was a typical assignmoit for the third 
week of the semester or a tocher's ""best or ''mc^ difficult^ assign- 
ment). Consequently, we decided only to request a copy of each sur- 
veyed teacher's syllabus (asking how much of the material in it was cov- 
eied in last year's dass) and a copy of the fixial examixtation. Even this 
lixnited information was difficult to obtain. Only about half of the sam- 
pled teachers were able to pxovide both items, because many do not re- 
tain syllabi and exaxnixiatioxis from one y^ to the next^ Requestixig; fi- 



^An aitenuiive strategy would be to ask te»che» about their cunent-vear courses. 
Untess such a survey were conducted in June, however, the respondents would ncA be able 
to report on topk coverage fc^ tfie entire year or provide a copy of tiie final examination* 
Another, tnore timMn^iming strategy would be to ask teactons to submit student 
£^gnments and examinaH^ns on a regular basis throu^umt the sdml year (e.g., once 
every three or four weeks). This strategy would produce, in ^ect, teacher portfolios 
containii^ accurate data on the cmrsewonc actually being delivered to students. Howevt^, 
such data collection would have to extend over most of an academic year. 



fi2 



ERIC 



-49^ 



nal examinations may also overestimate the extent to which teachers rely 
on the multiple-choice format In one of the seven schools in the SRA 
sample, we collected a sample of the tests that teachers administered 
throu^out the year, and we f6\md that students were more likely to be 
asked to elaborate the steps they used in solving mathematics problems 
or to answer social studies essay qu^ons on these tests than on Bnal 
examinations. Forty-seven percent of the 79 final examinations we re- 
viewed were multifde-choira tests, with mathematics and social studies 
examinations equally likely to be multiple<hoice. 

Despite the difficulty of col!e<iing data on course materials, such data 
provide one of the best ways to assess the corruptibility of indicator mea- 
sures. For example, teacher reports of using instructional strategic or 
topic coverage geared toward teadiing students higherorder skills can 
be diecked by examining the format and content of their examinations. 
Collecting data on major class projects might also be a way to ascertain 
whether such teaching is occurring, and it would be subject to fewer 
problems of int er pretation than routine assignments. 

Analysis of course materials (Le.^ the parts of textbooks that are actually 
covered in a class, major assignments, and examinations) is probably the 
most valid way to compare the curriculum actually taught with either 
professional standards or state curriculum frameworks; valid compar- 
isons often cannot be made with data reported at the topic leveL 
Analjrzing course materials can also help in determining the extent to 
whidt topic coverage, as reported by teachers, is dictated by the or^ni- 
zation of textbooks. Although course material reviews are expensive and 
time<onsuiring, they are key to the design of new indicators and to 
assessing the corruptibility of existing ones. Depending on a state's cur- 
riculum and textbook policies, course material analysis might also be a 
useful focus for special studies linked to the state's indicator system. 

At first glance, benchmark data may seem an unnecessary luxury for The Need for 
state indicator systems, and their collection an activity best left to aca- Benchmark Data 
demic researchers. However, we would argue that only with l^nchmark 
data can states obtain valid and useful information on student course- 
work, while still relying on relatively inexpensive data collection strate- 
gies such as teacher surveys. The deeper probes required to obtain 
benchmark data will allow states to validate their indicators over time 
and to identify where improved meastves are needed. Including 
benchmark data in a system of comsework indicators may also encour- 
age teachers to be more attentive in completing survey questionnaires 
because they will know that the state takes the information seriously 
enough to verify it through other sources. 
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... mdy with 
Iwnchmark data can 
states obtain valid 
and useful 
infomatton on 
sttident courseivork^ 
while relying on 
inexpensive data 
collection strategies. 



If s\a!tes dedde to close the cuxmit ga;^ in infomtation about student 
coiusework, they will have to balance the expensive data collection tra- 
ditionally associated with studying curriculum against the standardized, 
rdativdy ine^qpensive methods used for most indicators. A systan that 
relies prinuuily on teachor surveys and aggregate eiroUmextf statistics, 
supplemented by periodic benchmark studies, provides such a balance* 
We recommend that states willing to collect more defensive data on 
studwt ccursewOTk should make a signific^t and continuing invest- 
ment in benchmark data Sudi an investment would require spending 
about oneHjuarter of their coursework indicator budgets on these deeper 
probes. 
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6. NEXT STEPS FOR STATES 



Demands on state governments to generate more and better information about the quality 
of schooling have grown tremendously over the past decade As a result, hard choices have to be 
miuie about what data can be collected with limited resource Clearly, the first priority will con- 
tinue to be indicators of student performance. But a singular focus on schooling outcome l^ves 
key questions tuianswered. 



The need to know how learning opportunities vary ncro^ sch<x>ls and 
students, wl^ther polid^ to increase the rigor of student couiseworic 
have been translated into classroom practice, and how course content 
and instructional strategies affect student adiievonent argues for im- 
proved coursework indicators. The nation and individual states can set 
performance goals for their students, but whether those goals are met 
will depend largely on what is taught and how it is taught in individual 
schools and da^rooms. At present, states lack even the most basic in- 
formation about dtese critical elements of schooling. And, as the exam- 
ples in Ae preceding sections have illustrated, continued reliance on 
gross enrollment statistics enumerated by conventional course titles can 
only mislead policymakers and tl« public When courses called al^bra 
I, which most people would assume are reasonably ui\iform in content, 
can vary significantly from classroom to classroom, new measures of 
student coursework are needed. 

States must coitsider the arguments in fevor of improved onirsework 
indicators in lig^t of the absolute costs of deigning, a)llecting, analyz- 
ing, and reporting new data; the potential opportunity costs of not beii^ 
able to mount as <x>mplete an iiuiicator effort in some other area; and the 
additfonal burden that increased data collection would impose on 
schools and tochers. We believe that the boiefits to be gained from this 
information outweigh itm <x^, and that by careful design and execu- 
tion, these ojsts can be minimized. However, each state must assess its 
own future iitformation needs, its current data gaps, and the relative im- 
portance that policymakers and citizens as^gn to knowing what schools 
are teaching. 

For states &at do decide to invest in coursework indicators, we recom- 
mend a six-stage design process. This recommendation is based on our 
experience with the SRA project and other indicator development efforts. 
It is also derived from an overriding prixidple that we believe should 
gwde the design of all educational indicators: Indicator design in the 
service of moidtoring educational prog^%ss needs to balance bo^ techni- 
cal and policy considerations. In practice, this requires a design process 
that is research-based and poliqr-sensitive. The six steps outlined below 
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accommodate both sets of requiiements. The section concludes with a 
discussion of several issues that^ if not addressed early on, can generate 
prc^lems in the use of couisework indicators* 



THE DESIGN Sti^l. IdmHfytitestuU's unique information needs. Those needs will 

PROCESS determine not only what information is collected, but also how often it is 

collected, whether it is to include all schools or a sample, and to whom 
aiui in what format the data are to be reputed. A critical element of any 
indicator needs assessment is the identification of die mafor users of the 
information and the ways in which multiple audience can be served 
with the same data. In one sense, the primary audiences for coursework 
data are the general public (who will probably read the information in 
their newspapers), state-level elected officials, and education department 
administrators* However, if coursework data are not meaningful and 
us^ul to local educators and parents, the information is not likely to be 
used by them^ nor are they apt to regard its collection seriously. To serve 
these multiple audience, data must be reported in difierent formats and 
levels of detail But the need to serve diverse audiences also influences 
how mea^ires are defined, what data collection strategies are uwd (e.g., 
state^level audiences may need information only from a representative 
sample of schools, but such information would be of little use to individ- 
ual schools), and the frequency and timing of reports. 

Stage 2. Review reant research on indicator design. The growing body 
of work on indicator design includes analyses of the major studies of 
schooling and what they imply for the design of improved coiirsework 
indicators (Mumane and Raizen, 1988; Oakes and Carey, 1989). This re- 
search explores in detail the design standards outlined in Section 2, ana- 
lyzes the technical reliability and validity issues that must be addressed, 
and assesses existing indicators on those criteria. 

If states decide to collect indicator data on the elementary school curricu- 
lum as well, this is the s^ge at which they would need to think about 
how th(^ measures and the data collection strategies that underlie them 
would differ from a coursework focus at the high school level Although 
we have not examined this issue in any depth, we ore quite certain that it 
would be necessary to use a different conceptualization and measure- 
ment process for elementaiy-level curriculum indicators than the one we 
used for high school coursework. Currioilar content in elementary 
schooling is not as dearly divided into separate coirn^ as it is in high 
schools, and it would be necessary to look for subject-matter rontent 
within the contort of other subjects (e.g*, reading content within social 
studies, science within reading). Also, in contrast to high schools, where 
an entire class or section often consists of students in the same ability 
group, elementary classes typically contain students of different ability 
levels who are then organized by those ability levels into work groups 
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within the classroom. These instructional groups, several of which op 
erate in each dassroom, are the primary unit of analysis for curriculum 
studio because teachers tailor much of their instruction to these groups 
(Barr and Dreeben, 1983), Therefore, elementary teachere could not 
validly report on topic coverage or instructional strategies in the single 
format used for high school teachers. They would have to report about 
each instructional group separately or try to determine what proportion 
of their entire class was exposed to different topics. Hnally, elementary 
students are much less likely than high school students to be a n'^liable 
source of information about the instruction they receive. Concerns about 
access to learning opportunities and the quality of those opportumties 
call for curriculum indicators at both the elementary and secondary 
levels^ and both require that techniral criteria and policy requirements be 
balanced throughout the design pro«ss. But the conceptualization and 
measurement issues are quite different for each level of schooling, and 
the schooling research that should guide the development process is also 
different for each. 

Stage 3. Refifte and adapt existing indicators. In some areas, such as 
mathematics topic coverage, instmctional activities, and teacher qualifi- 
cations, states can adapt measures used in the SRA project and studies 
such as the NELS, NAEP, and SIMS. These measures are not without 
flaws, but most of them can be remedied with only minor modifications. 
However, even the mmt robust measures must be adapted to the state 
context in which they are used. It is necessary to take into consideration 
which state policies influence cour^work, how schools are or^nized in 
a state (e.g., the extent of state as compared with local direction in cur- 
ricular content, the grade levels at which specific courses are offered or 
required), and the range of variation in students and classrooms across 
the state. 

Existing coursework measures must also be adapted if they are to be in- 
tegrated into ongoing information systems. In most states, this means 
linking coursework data with student assessments and with the collec- 
tion of data on other elements of schooling, such as resource levels and 
student and teacher characteristics. The integration of coursework indi- 
cators into existing systems will depend on the extent to which they have 
been deBned in ways that draw on other data sources or that piggyback 
on other routine data collection. For example, it may be possible to ob- 
tain some data for an indicator of the match between teacher assign- 
ments and qualifications from the state teacher licensure agency. 
Similarly, states might arrange for teachers to complete coursework sur- 
veys while their students are taking the statewide assessment. 

Stage 4. Develop new measures. For those areas of coursework where 
little design work has been done (in other words, subject areas outside 
mathematics), new measwes will have to be developed. This process is 
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likely to take two years for each subject area- Textbooks^ curriculum 
frameworks, and profes^onal standards will have to be reviewed, and 
in-depth interviews with department chairs and teaches in a variety of 
schools will be necessary to idmtify the full range of couise objectives, 
topics, and instructional strategies, and to determine how they are dis- 
treated aao^ di^rait types of courses and student ability levels. 
Finally, consensus panels of teachers and subject-matter experts will 
have to be convened to ensure that measuies are consistent with practi- 
tionet and expert judgment, and to select candidate indicators for inclu- 
sion on teacher surveys. 

There are two strategies that might be used to reduro the development 
cwts in any ^gle state and to spread thc^ costs over a longer time pe- 
riod* Even thoiigh indicators must be adapted to the information needs 
and policy context of individual states, much of the initial development 
work will be ^milar. Therefore, states might consider forming consortia 
(as they are already doing in the assessment area) to collaborate on de- 
veloping new coursework indicators* Within a single state, development 
costs can be spread out over several years by focusing on only cme or two 
subject areas at a time- The order in which development occurs might be 
based on the subject areas emphasized in a state's performance goals or 
in its curriculum polid^. 

m 

Stage 5. Devise strategies for piloting tite teacher surueys and collecting 
ben^mark data. For the most part, this is a straightforward task, but it 
does require that decisions be made about the kinds of benchnuurk data 
to be collected, collection frequency, and how the data will be used. For 
example, will benchmaric data be a>llected periodically only to validate 
teacher survey data and to develop new indicators? Or will it be the 
sub^ of fiill-blown studies that complement routine indicator data but 
stand on their own with independent conclusions? 

Stage 6, Implement data collection* If pist experience is any guide, an 
iterative process involving several rounds of data collection followed by 
revisions in the measures and instruments will be needed to make the re- 
siilting indicators technically valid and und^tandable to their users; 
this process will also be needed to establish that data collection proce- 
dures are smooth and cost^-efftdent Therefore, states that dedde to in- 
vest in improved coursework measures will have to remain committed 
for the long haul A ''one-shor collection of student coursework data 
will not produce information worth the effort. 



AVOIDING Those who comnussion and design new indicators can rarely control the 

PROBLEMS ways in which tho^ indicators are used. Nevertheless, taking into ac- 

coimt how similar measure have been used in the past and antidpating 
how new ones nright be used can help avoid problems. The most serious 
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di££culti^ stem £rom :wo simple facts about indtoitoxs, nxst, as they 
^in si^iificance in the policy wwld^ ixidicatois provide less valid infor- 
mation aboirt what is actually oocunrfaig in ckesrooms— that is, th^ be- 
come corrupted as n^sur^ Second, indicators are powerful levers for 
dianging dassroom behavior, but that change often is not what policy- 
maikers and indicator designers intended 

Both of these probleins are evident in rtudent assessnaent systems Tests 
may no long^ measure actual adiiev^nent because students have been 
coadied on test content;, and the curriculum to which they are exposed 
has been narrowed to reflect ttie lowo^levd skills typicaUy tested Such 
unintended consequences are due not so much to the tedmical quality of 
assessments as to how results are used. As states have placed more em* 
phasis on student test scores and have ^tadied major policy actions to 
the results, tiiese 'Idgh-stakes'' conditions have changed the meaning 
and effect of assessment data. 

It is unlikely that coiusework data could ever have the same kind of hi^ 
stakes attadted to their use. But ih^ are situaticms in which indicators 
might beccnne corrupted or teaching behavior might change in undesir- 
able ways. We have already sug^sted that history teachers in the SRA 
sample may have imderestimated the frequency with which they lec- 
tured or assigned students texd>ook reading in dass because tih activi- 
ties have been deemphasized in current reform rhetoric. Similarly, 
teach^ might tend to report a greater anphasis on course objectives 
that happen tobein vogue atagiven time. Neither of these examples is 
particularly serious if a system fbr collecting benchmark data is in place. 
Teacher reports about instructional activities can be compared with class 
assigrunents, and even within the survey data itself, reports of course 
ob^tives can be assessed for consistency with topic coverage and in- 
structional activity reports. 

Potentially more serious problems could result if coursework informa- 
tion became another basis on which schools were compared publidy. 
Just as student test scores are often interpreted out of context comparing 
schools on the basis of how much a>ntent is covered in different courses 
or whether instructional strategies are consistent with current reform 
rhetoric could le^ to greater standardization of instructional practices. 
This possibility represents a very real dileirmia for poUcyriuikers. A ma- 
jor reason for collecting a>ursework date is to ^isure that coiuse a>ntent 
is consistent with pubUc and professiorml expectetions and that learning 
opportunities are relatively equal across schools and students. 
Therefore, stete policymakers may want to send a strong signal that they 
and the public do not want to have algebra classes spending mixt of 
their time on ariAmetic topics, or that they expect VS. history students 
to be given the opportunity to read priirtary materials and write resrarch 
papers. The intended effect of that signal would be for educators to 
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change their classroom behavior But few policyxnakeis would want 
teachers to have no flexibility in exercising their professional judgment 
about which instructional strategies are most effective for their own stu- 
dents. The line between expecting research papers to be assigned and an 
unwarranted standardization of practice can become blurred if course- 
work data are xised in inappropriate ways (e.g*, to evaluate schools or 
individual teachers). 

Such problems can be minimized if state officials are clear from the very 
beginning about the appropriate uses of couisework indicators. These 
data are most useful for documenting trends, assessing the performance 
of the overall educational system, identifying and tar^ting policy solu- 
tions, and aiding in local improv^nent efforts. If they are used to com- 
pare schools or to assess teacher performance, they will cease to provide 
valid information and may even distort instructional practices in ways 
that compromiw student learning. 

CONCXUSIONS Improving the information base on which decisions about educational 

policy and practice are made is not easy. Designing valid and useful 
m^sures takes time; collecting the necessary data imjx^es financial and 
other costs; and using indicator data in a constructive way requires con- 
siderable thought and planning. While these represent formidable chal* 
lenges in any type of indicator development, (hey are particularly de- 
manciing in the coursework area because of the need for validation and 
detailed benchmark data. 

Despite these challenges, the price that stales will pay for not collecting 
better coiuisework data may even higher. Strategies for attaining per- 
formance goals cannot be implemented, classroom learning opportuni- 
ties cannot be equalized, and policy impacts omnot be effectively moni- 
tored if current information gaps are not filled. Indicator development 
will never have the viability or political appeal of new policy initiatives 
aimed at improving schools. Nevertheless, data about what schools are 
teaching form the cornerstone of those policies, and the constituents of 
public sch(x>ling need that information to continue the work of educa- 
tional reform. 
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Appendix 
SRA SAMPLE 



Number of schools 7 

California 

Urban 2 

Suburban 1 

Rural 1 

Geoipa 

Suburban 1 

Rural 2 

Enrollment at smallest school 336 

Enrollment at laigest three schools 2,000 

Number of teachers surveyed 

Mathematics 73 

Social studies • ^ 

Response rate (percent) * 92 

Number of sections in the Hve course categories exanuned 

Algebra I and below - 86 

Algebra n 18 

U.S. History • 38 

U.S. Government 28 

Number of students surveyed 

Tenth grade 2^71 

Twelfth grade 1^937 

Response rate (percent) 75 

Number of transcripts coded^ 

Class of 1986/87^ 511 

Class of 1989 514 

Class of 1991 516 



^vcflty-five transcripis were sampled from the relevant ninth grwSe ctass at each school* 
but about 2 percent were del^ because the transcripts eitho* wm sampled from the wrong 
classes or were not photocopi^ in their ottirety. 

^The class of 19% was used for the California schools and the class ofl9S7 was used for 
the Geoigia schools because they repftsem the last classes to progress through high school in 
each state before state-mandated increases in course ftqulremenu took effect 



71 



REFERENCES 



Ban-, R., & Dreeben, R. (1983). Hmv schools xvork. Chicago: University of Chicago Press. 

Bennett W.J. (1988). American education: Making it work. Washington, IX: U5. Department of 
Education. 

Blank, R. K. (1988). State data amUidfle on indicators of science/math aiucation. Draft paper. 
Washington, DO Council of Chief State School Officers. 

Blanl^ R. IC, & Schilder, D. (1989). State-bjf-state indicators of science and mathematics education: 
preliminary report, Washingtoiv DC: Coundlof Chief State School Officers. 

Bureau of R^eanh and Assessment (1986). The Mgfi school experience in Massachusetts 
(Publication «14438-74->500-5-86). MA: Department of Education. 

Burstein, L (Ed.) (Forthcoming). The lEA study of mathematics III: Student growth and classroom 
processes in lower setmdary sdtools. London: Pergamon. 

Burstdn, L, Aschbacher, P., Chen, Z, Un, L., & Sen, Q. (1986). Establishing the content validity of 
teste dmgmd to serve multiple purposes: Bridging secondary-postsecondary mathematics. Los 
Angeles: UCLA Center for the Study of Evaluation. 

Buratein, L, Chen Z, & Kim, K-S. (1989). Analysis of procedures for asse^ing content coverage and its 
effais on student achieuemcnt. Lc» Angeles: UCLA Center for the Study of Evaluation. 

Oune, W. R, White, P., & Patterson, J. (1989). The implementation and effects of high school 
graduation requirements: First steps toward curricular reform. New Brunswidk, NJ: Center for 
Policy Research in Education. 

Darling-Hanunond, L., & Hudson, L. (1989). Teachers and teaching. In R. J. Shavelson, L. M. 
McDonnell, & }. Oakes (Eds.). Indicators for monitoring mathematics and science education: A 
sourc^Hfok. (pp. 66-95). Santa Monica, CA: The RAND Corporation. 

Dossey, J. A., Mullis, L V. S., Lindquist, M. S., & Chambers, D. (1988, June). The mathematics report 
card. Are we measuring up? Trends and achievement based on the 1986 national assessment (Report 
NO.17-M-01). Princeton, NJ: Educational Testing Service. 

Gamoran, A. (1987). Instructional and institutional effects of ability grouping. Sociology of 
Education, 59, 185-198. 

Gamoran, A., & Berends, M (1987). The effects of stratification in secondary schools: Synthesis 
of survey and ethnographic research. Review of Educational Research, 52 (4), 415-435. 

Goertz, M. E. (1989). Course-taking patterns in the 1980s. New Brunswick, NJ: Center for Policy 
Research in Education. 

Hanson, T. L. (1989). Curricular change in Dade county 1982-83 to 1986-87: A replication of the 
PACE study. New Bnmswick, NJ: Center for Policy Research in Education. 

Jones, L. V., Davenport, E. C, Bryson, A., Bckhuis, T., & Zwick, R. (1986). Mathematics and 
science test scores as related to courses taken in high school and other factors. Journal of 
Educational Measurement, 23 (3), 197-208. 

MacRae, D., Jr. (1985). Policy indicators. Chapel Hill, NC: University of North Carolina Press. 



-60- 



McDonnell, L. M. (1989). The policy context In R. J. Shavelson, L. M McDonnell & J. Oakes 
(Eds.). Indicators for monitoring mathmatits and sciemx education: A sourcebook, (pp. 241-269). 
Santa Monica, CA: The RAND Corporation. 

McKnighl, C. C, Crosswhite, F. J., Dossey, J. A., Kifer, E., Swafford, S. O., Travere, K. J., & 
Coon^, T. J. (1987). The underachkoing curriculum: Assessing U.S. mathematics from an 
intemiional pers pect ive. Champaign, IL: Stipes Publishing. 

Mumane, R J., & Raizen, S. A (1^). Imprmmg indicators of the quality of science and mathematics 
education in pides K-12. Washington, DC National Academy Press. 

National Council of Teachers of Mathematics. (1989). Curriculum and eoaluation standards for 
school nuUhenuiti(s. Reston,VA Worldng Grouj^ of the Cbmmis^on on Standards for School 
Mathematics of the National Coimdl of Teachers of Mathematics. 

Oakes J. (1985). Keeping track How schoob structure inequality. New Haven, CT: Yale University 
Press. 

Oakes, J. (1986). Educational indicators: A guide pr p(Aiqfmakers. New Brunswick, NJ: Center for 
Policy Research in Education. 

Oakes, J., & Carey, N. (1989). Curriculum In R J. Shavelson, L. M. McDonnell, & J. Oakes (Eds.). 
Indicators for monitoring mathemati^ and sciemx education: A sourc^Kwk. (pp. 96-122). Santa 
Monica, CA: The RAND Corporation. 

Oakes, J., Ormseth, T., Bell R, & Camp, P. (1990). Unequal opportunities: The effects of race, social 
class, and ability grouping on access to sdence and mathematics education. Santa Monica, CA: The 
RAND Corporation. 

OERI State Accoimtability Study Group (1988). Creating responsible and responsive accountability 
systen^. Washington, DC: U.S. Department of Education. 

Raizen, S. A, & Jones, L. V. (Eds.). (1985). Indicators of precollege education in science and 
mathematics: A preliminary review. Washington, D.C.: National Academy Press. 

Shavelson, R ]., McDoiuiell, L. M., Oakes, J. (Eds.). (1989). Indicators for monitoring mathematics' 
and science education: A sourcebook. &nta Monica, CA: The RAND Corporation. 



ERIC 



73 



